Abstract
Automatic speech recognition (ASR) has benefited from advances in pretrained speech and language models, yet most systems remain constrained to monolingual settings and short, isolated utterances. While recent efforts in context-aware ASR show promise, two key challenges persist: limited multilingual support and the absence of principled alignment between speech and contextual representations. In this paper, we introduce a context-aware multilingual ASR framework that supports diverse languages and accents while preserving the modularity of pretrained models. Our approach combines a frozen speech encoder and a decoder-only language model via a lightweight projection module, allowing structured context prompts, including dialogue history and biasing words, to guide transcription. To improve interaction between speech and context, we employ a contrastive learning objective that aligns their representations in a shared embedding space. Evaluations on over 1,500 hours of real-world conversational speech across 11 languages and 5 English dialects show that contextual input consistently improves recognition quality. Contrastive alignment provides additional gains when applied to different context types, with an overall performance gain of over 5%. These results highlight the importance of both contextual modeling and cross-modal alignment in multilingual ASR.
Abstract (translated)
自动语音识别（ASR）技术受益于预训练的语音和语言模型的进步，但大多数系统仍局限于单语环境和简短孤立的话语。虽然最近在上下文感知ASR方面的努力显示出前景，但仍存在两个关键挑战：对多语言支持有限以及缺乏将语音与上下文表示进行原则性对齐的方法。本文介绍了一种新的上下文感知的多语言ASR框架，该框架支持多种语言和口音的同时保留了预训练模型的模块化特性。我们的方法结合了一个冻结的语音编码器和一个解码器主导的语言模型通过轻量级投影模块连接起来，这使得结构化的上下文提示（包括对话历史记录和倾向词）能够引导转录过程。为了改善语音与上下文之间的交互，我们采用了一种对比学习目标，该目标在共享嵌入空间中对齐它们的表示形式。 通过对跨越11种语言及5种英语方言超过1,500小时的真实会话语音进行评估显示，上下文输入一致提高了识别质量。当应用于不同类型的情况下，对比对齐提供了额外的优势，并且整体性能提高了超过5%。这些结果强调了多语言ASR中上下文建模和跨模式对齐的重要性。
URL
https://arxiv.org/abs/2603.06505
