Abstract
Audio-Visual Speech Recognition (AVSR) integrates acoustic and visual information to enhance robustness in adverse acoustic conditions. Recent advances in Large Language Models (LLMs) have yielded competitive automatic speech recognition performance and shown effectiveness for AVSR. However, prior approaches project audio and visual features independently or apply shallow fusion, limiting cross-modal alignment and complementary exchange while increasing the LLM's computational load. To address this, we propose AVUR-LLM, an LLM-based Audio-Visual Speech Recognition via Sparse Modality Alignment and Visual Unit-Guided Refinement. Experiments on LRS3 demonstrate state-of-the-art results for AVSR. Under additive-noise conditions at 0 dB SNR, it achieves 37% relative improvement over the baseline system.
Abstract (translated)
音频-视觉语音识别（AVSR）结合了声学和视觉信息，以增强在不良声学条件下的鲁棒性。近年来，在大型语言模型（LLMs）方面的进展已经产生了具有竞争力的自动语音识别性能，并且对于AVSR也显示出有效性。然而，之前的方法独立地投影音频和视觉特征或将浅层融合应用到它们之上，这限制了跨模态对齐和互补交换，并增加了LLM的计算负担。为了解决这个问题，我们提出了AVUR-LLM，这是一种基于大型语言模型（LLMs）的语音识别方法，通过稀疏模式对齐和视觉单元引导优化来实现音频-视觉语音识别。 在LRS3数据集上的实验显示，该方法达到了最先进的AVSR结果，在0 dB信噪比下的加性噪声条件下，与基线系统相比，相对改进达到了37%。
URL
https://arxiv.org/abs/2603.03811
