Abstract
Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.
Abstract (translated)
基于标准大语言模型的语音识别系统通常孤立处理话语，限制了其利用对话上下文的能力。本研究探讨了多模态的先前轮次上下文是否能提升基于大语言模型的自动语音识别效果，以及如何高效表征该上下文。我们发现，经过监督式多轮训练后，对话上下文主要有助于识别上下文相关实体。然而，直接使用原始上下文成本高昂，因为先前轮次的音频标记序列会随对话长度快速增长。为此，我们提出抽象压缩方法，用固定数量的学习隐式标记替换先前轮次的音频部分，同时显式保留对应文本转写。在同领域和跨领域测试集上，压缩模型以更小的先前轮次音频占用空间，部分恢复了原始上下文条件设置带来的性能提升。我们还针对压缩配置及其权衡进行了针对性分析。
URL
https://arxiv.org/abs/2603.26246
