Abstract
Despite extensions to speech inputs, effectively leveraging the rich knowledge and contextual understanding of large language models (LLMs) in automatic speech recognition (ASR) remains non-trivial, as the task primarily involves direct speech-to-text mapping. To address this, this paper proposes chain-of-thought ASR (CoT-ASR), which constructs a reasoning chain that enables LLMs to first analyze the input speech and generate contextual analysis, thereby fully exploiting their generative capabilities. With this contextual reasoning, CoT-ASR then performs more informed speech recognition and completes both reasoning and transcription in a single pass. Moreover, CoT-ASR naturally supports user-guided transcription: while designed to self-generate reasoning, it can also seamlessly incorporate user-provided context to guide transcription, further extending ASR functionality. To reduce the modality gap, this paper introduces a CTC-guided Modality Adapter, which uses CTC non-blank token probabilities to weight LLM embeddings, efficiently aligning speech encoder outputs with the LLM's textual latent space. Experiments show that, compared to standard LLM-based ASR, CoT-ASR achieves a relative reduction of 8.7% in word error rate (WER) and 16.9% in entity error rate (EER).
Abstract (translated)
尽管语音输入已得到扩展，但在自动语音识别（ASR）中有效利用大语言模型（LLMs）丰富的知识与上下文理解能力仍非易事，因为该任务主要涉及直接的语音到文本映射。为此，本文提出思维链语音识别（CoT-ASR），该方法构建推理链使LLM能够首先分析输入语音并生成上下文分析，从而充分发挥其生成能力。基于此上下文推理，CoT-ASR随后执行更具依据的语音识别，并在单次过程中完成推理与转写。此外，CoT-ASR天然支持用户引导式转写：虽然设计上由模型自主生成推理，但也可无缝整合用户提供的上下文以引导转写，进一步拓展ASR功能。为减少模态差异，本文引入CTC引导的模态适配器，该适配器利用CTC非空白标记概率对大语言模型嵌入进行加权，高效对齐语音编码器输出与LLM的文本潜在空间。实验表明，相较于标准基于LLM的ASR，CoT-ASR在词错误率（WER）上实现相对8.7%的降低，在实体错误率（EER）上实现相对16.9%的降低。
URL
https://arxiv.org/abs/2604.00610
