Abstract
Audio-Visual Speech Recognition (AVSR) has achieved remarkable progress in offline conditions, yet its robustness in real-world video conferencing (VC) remains largely unexplored. This paper presents the first systematic evaluation of state-of-the-art AVSR models across mainstream VC platforms, revealing severe performance degradation caused by transmission distortions and spontaneous human hyper-expression. To address this gap, we construct \textbf{MLD-VC}, the first multimodal dataset tailored for VC, comprising 31 speakers, 22.79 hours of audio-visual data, and explicit use of the Lombard effect to enhance human hyper-expression. Through comprehensive analysis, we find that speech enhancement algorithms are the primary source of distribution shift, which alters the first and second formants of audio. Interestingly, we find that the distribution shift induced by the Lombard effect closely resembles that introduced by speech enhancement, which explains why models trained on Lombard data exhibit greater robustness in VC. Fine-tuning AVSR models on MLD-VC mitigates this issue, achieving an average 17.5% reduction in CER across several VC platforms. Our findings and dataset provide a foundation for developing more robust and generalizable AVSR systems in real-world video conferencing. MLD-VC is available at this https URL.
Abstract (translated)
音视频语音识别（AVSR）在离线条件下已取得显著进展，但其在真实视频会议（VC）场景中的鲁棒性尚缺乏系统研究。本文首次对主流VC平台上的先进AVSR模型进行了系统评估，揭示了传输失真与人类夸张自发表达导致的严重性能下降。为填补此空白，我们构建了首个面向视频会议场景的多模态数据集\textbf{MLD-VC}，包含31位说话人、22.79小时音视频数据，并明确引入伦巴德效应以增强人类夸张表达。通过全面分析发现，语音增强算法是导致分布偏移的主要来源，该偏移改变了音频的一、二阶共振峰。有趣的是，伦巴德效应引发的分布偏移与语音增强引入的偏移高度相似，这解释了为何在伦巴德数据上训练的模型在视频会议中表现出更强鲁棒性。基于MLD-VC对AVSR模型进行微调可缓解此问题，在多个VC平台上平均降低17.5%的字错误率。本研究结果与数据集为开发更鲁棒、泛化能力更强的真实视频会议音视频语音识别系统奠定了基础。MLD-VC数据集可通过此超链接获取。
URL
https://arxiv.org/abs/2603.22915
