Abstract
Contextual automatic speech recognition (ASR) with Speech-LLMs is typically trained with oracle conversation history, but relies on error-prone history at inference, causing a train-test mismatch in the context channel that we term contextual exposure bias. We propose a unified training framework to improve robustness under realistic histories: (i) Teacher Error Knowledge by using Whisper large-v3 hypotheses as training-time history, (ii) Context Dropout to regularize over-reliance on history, and (iii) Direct Preference Optimization (DPO) on curated failure cases. Experiments on TED-LIUM 3 (in-domain) and zero-shot LibriSpeech (out-of-domain) show consistent gains under predicted-history decoding. With a two-utterance history as context, SFT with Whisper hypotheses reduce WER from 5.59% (oracle-history training) to 5.47%, and DPO further improves to 5.17%. Under irrelevant-context attacks, DPO yields the smallest degradation (5.17% -> 5.63%), indicating improved robustness to misleading context. Our code and models are published on this https URL.
Abstract (translated)
基于语音大语言模型（Speech-LLM）的上下文自动语音识别（ASR）通常使用真实对话历史（oracle history）进行训练，但在推理时却依赖于易出错的预测历史，导致上下文通道中存在训练-测试不匹配问题，我们称之为上下文暴露偏差（contextual exposure bias）。我们提出一个统一训练框架以提升模型在真实历史条件下的鲁棒性：（i）教师错误知识（Teacher Error Knowledge），即使用Whisper large-v3的假设文本作为训练时的历史输入；（ii）上下文丢弃（Context Dropout），用于正则化模型对历史的过度依赖；（iii）对精选的失败案例进行直接偏好优化（DPO）。在TED-LIUM 3（领域内）和零样本LibriSpeech（领域外）上的实验表明，在预测历史解码条件下该方法持续取得增益。当使用两句历史作为上下文时，基于Whisper假设进行监督微调（SFT）可将词错误率（WER）从5.59%（真实历史训练）降至5.47%，而DPO进一步优化至5.17%。在无关上下文攻击场景下，DPO产生的性能退化最小（5.17% → 5.63%），表明其对误导性上下文具有更强的鲁棒性。代码与模型已公开发布于此https链接。
URL
https://arxiv.org/abs/2603.24034
