Abstract
Recent studies have demonstrated that incorporating auxiliary information, such as speaker voiceprint or visual cues, can substantially improve Speech Enhancement (SE) performance. However, single-channel methods often yield suboptimal results in low signal-to-noise ratio (SNR) conditions, when there is high reverberation, or in complex scenarios involving dynamic speakers, overlapping speech, or non-stationary noise. To address these issues, we propose a novel Visual-Informed Neural Beamforming Network (VI-NBFNet), which integrates microphone array signal processing and deep neural networks (DNNs) using multimodal input features. The proposed network leverages a pretrained visual speech recognition model to extract lip movements as input features, which serve for voice activity detection (VAD) and target speaker identification. The system is intended to handle both static and moving speakers by introducing a supervised end-to-end beamforming framework equipped with an attention mechanism. The experimental results demonstrated that the proposed audiovisual system has achieved better SE performance and robustness for both stationary and dynamic speaker scenarios, compared to several baseline methods.
Abstract (translated)
最近的研究表明，整合辅助信息（如说话人的声纹或视觉线索）可以显著提高语音增强(SE)的性能。然而，在低信噪比(SNR)条件、存在高混响情况或是涉及动态发言者、重叠讲话以及非平稳噪声等复杂场景下，单通道方法通常会导致次优结果。为解决这些问题，我们提出了一种新颖的视觉引导神经波束成形网络(VI-NBFNet)，该网络将麦克风阵列信号处理与深度神经网络(DNN)相结合，并利用多模态输入特征进行操作。 所提出的网络采用了预训练的视觉语音识别模型来提取唇部运动作为输入特征，这些特征用于语音活动检测(VAD)和目标发言者识别。通过引入一种配备注意力机制的监督端到端波束成形框架，该系统旨在处理静态和移动发言者的场景。实验结果表明，在与几种基线方法相比时，所提出的视听系统在静止和动态发言者的场景下均实现了更好的语音增强性能和鲁棒性。
URL
https://arxiv.org/abs/2603.05270
