Abstract
Recently, Automatic Speech Recognition (ASR) systems (e.g., Whisper) have achieved remarkable accuracy improvements but remain highly sensitive to real-world unseen data (data with large distribution shifts), including noisy environments and diverse accents. To address this issue, test-time adaptation (TTA) has shown great potential in improving the model adaptability at inference time without ground-truth labels, and existing TTA methods often rely on pseudo-labeling or entropy minimization. However, by treating model confidence as a learning signal, these methods may reinforce high-confidence errors, leading to confirmation bias that undermines adaptation. To overcome these limitations, we present ASR-TRA, a novel Test-time Reinforcement Adaptation framework inspired by causal intervention. More precisely, our method introduces a learnable decoder prompt and utilizes temperature-controlled stochastic decoding to generate diverse transcription candidates. These are scored by a reward model that measures audio-text semantic alignment, and the resulting feedback is used to update both model and prompt parameters via reinforcement learning. Comprehensive experiments on LibriSpeech with synthetic noise and L2 Arctic accented English datasets demonstrate that our method achieves higher accuracy while maintaining lower latency than existing TTA baselines. Ablation studies further confirm the effectiveness of combining audio and language-based rewards, highlighting our method's enhanced stability and interpretability. Overall, our approach provides a practical and robust solution for deploying ASR systems in challenging real-world conditions.
Abstract (translated)
最近，自动语音识别（ASR）系统（例如Whisper）在准确性方面取得了显著的改进，但仍然对现实世界中的未见数据（具有较大分布变化的数据），如噪音环境和多样化的口音非常敏感。为了解决这个问题，在推断时间进行测试适应性调整（TTA）展示了极大的潜力，可以在没有真实标签的情况下提高模型的适应性。现有的TTA方法通常依赖于伪标记或熵最小化。然而，通过将模型置信度视为学习信号，这些方法可能会强化高置信度错误，导致确认偏差从而削弱了适应能力。 为了克服这些限制，我们提出了ASR-TRA，这是一种受因果干预启发的新型测试时间增强适应框架。具体而言，我们的方法引入了一个可学习的解码器提示，并利用温度控制的随机解码来生成多样化的转录候选方案。通过一个奖励模型对音频和文本语义一致性进行评分，这些候选方案被评估并反馈用于通过强化学习更新模型参数和提示参数。在LibriSpeech合成噪音数据集以及L2 Arctic口音英语数据集中进行了全面实验，结果表明我们的方法相较于现有的TTA基线，在保持较低延迟的情况下实现了更高的准确性。 消融研究进一步证实了结合音频和语言奖励的有效性，突显出我们方法增强了稳定性和可解释性。总体而言，我们的方法为在具有挑战性的现实条件下部署ASR系统提供了实用且稳健的解决方案。
URL
https://arxiv.org/abs/2603.05231
