Abstract
Long-form speech recognition with large encoder-decoder models such as Whisper often exhibit hallucinations, repetition loops, and content omissions. These errors can accumulate and be further amplified when the previous segment's transcription is used as decoding context. We propose Whisper-CD, a training-free contrastive decoding framework that contrasts clean-audio logits against negative logits computed from three acoustically motivated perturbations: Gaussian noise injection, silence signal, and audio temporal shift. We aggregate these negatives via the log-sum-exp operator, building a unified multi-negative objective for token-by-token decoding. Across five English long-form benchmarks, Whisper-CD reduces WER by up to 24.3pp on CORAAL and shows 48% faster token generation throughput than beam search. Because Whisper-CD operates purely at inference time, it can be applied as a drop-in replacement to already-deployed Whisper systems without retraining.
Abstract (translated)
长语音识别中，使用如Whisper这样的大型编码器解码器模型时常会出现幻觉、重复循环和内容遗漏等问题。这些问题在前一段转录被用作解码上下文时会积累并进一步放大。我们提出了一种无需训练的对比解码框架——Whisper-CD，该框架通过将清洁音频的概率向量与三种声学动机的扰动计算出的负样本概率向量进行对比来减少错误：高斯噪声注入、静音信号以及音频时间偏移。这些负样本通过对数和指数运算聚合在一起，形成了一种统一的多负样本目标函数，用于逐令牌解码过程。 在五个英文长语音基准数据集上，Whisper-CD将WER（单词错误率）降低了最多24.3个百分点（CORAAL数据集中），并且比束搜索方法快48%。由于Whisper-CD仅在推理阶段工作，因此可以在不重新训练的情况下直接替换已部署的Whisper系统。
URL
https://arxiv.org/abs/2603.06193
