Abstract
Audio-visual speech recognition (AVSR) is an extension of ASR that incorporates visual signals. Current AVSR approaches primarily focus on lip motion, largely overlooking rich context present in the video such as speaking scene and on-screen text. To tackle such CAVSR (AVSR including rich visual Context), we propose VASR designed to "see" and reason the visual context to improve speech recognition. Specifically, we construct an Audio-Visual Chain-of-Thought (AV-CoT) that explicitly enforces intermediate cross-modal grounding between acoustic signals and visual evidence. This evidence-driven reasoning mitigates the "single-modality dominance" problem, where models either over-rely on visual context or fail to utilize it. Besides, to address the data scarcity, we construct and release a corresponding data pipeline and test set. Experiments show that AV-CoT effectively mitigates the single-modality dominance, achieving state-of-the-art performance in CAVSR. The project is open-sourced.
Abstract (translated)
音频-视觉语音识别（AVSR）是自动语音识别（ASR）的扩展，它结合了视觉信号。目前的AVSR方法主要关注唇部运动，而忽略了视频中存在的丰富背景信息，如说话场景和屏幕上的文本。为了应对包含丰富视觉上下文（CAVSR）的音频-视觉语音识别挑战，我们提出了一种名为VASR的方法，该方法旨在“看到”并推理视觉上下文以提高语音识别性能。具体来说，我们构建了一个音频-视觉链式思考（AV-CoT），它明确地强制执行声学信号与视觉证据之间的中间跨模态基础工作。这种基于证据的推理减轻了“单一模态主导”的问题，即模型要么过度依赖视觉上下文，要么未能有效利用它。 此外，为了解决数据稀缺的问题，我们构建并发布了相应的数据管道和测试集。实验表明，AV-CoT有效地缓解了单一模态主导的问题，在CAVSR方面取得了最先进的性能表现。该项目已经开源。
URL
https://arxiv.org/abs/2603.07263
