Abstract
Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework using Shapley values to analyze modality contributions in AVSR. Through experiments on six models across two benchmarks and varying SNR levels, we introduce three analyses: Global SHAP for overall modality balance, Generative SHAP for contribution dynamics during decoding, and Temporal Alignment SHAP for input-output correspondence. Our findings reveal that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation. Modality balance evolves during generation, temporal alignment holds under noise, and SNR is the dominant factor driving modality weighting. These findings expose a persistent audio bias, motivating ad-hoc modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.
Abstract (translated)
音频-视觉语音识别（AVSR）利用声学和视觉信息，在噪声环境下进行稳健的语音识别。然而，模型如何平衡这些模态仍不清楚。我们提出了Dr. SHAP-AV框架，该框架使用Shapley值来分析AVSR中模式贡献情况。通过在两个基准测试集上对六种不同模型以及各种信噪比（SNR）水平进行实验，我们引入了三种分析方法：全局SHAP用于整体模态平衡、生成式SHAP用于解码过程中的贡献动态变化、时间对齐SHAP用于输入输出对应关系。我们的研究发现表明，在噪声环境下，模型倾向于依赖视觉信息，但在严重退化情况下仍保持较高的声学信息贡献水平。模式间的平衡在生成过程中发生变化，而时间上的对齐性在噪声下依然存在，信噪比是驱动模式权重的主要因素。 这些发现揭示了持续存在的音频偏见，这促使我们需要采用临时的模式权重机制，并将基于Shapley值的归因方法作为AVSR诊断的标准。
URL
https://arxiv.org/abs/2603.12046
