Abstract
Visual speech recognition is a technique to identify spoken content in silent speech videos, which has raised significant attention in recent years. Advancements in data-driven deep learning methods have significantly improved both the speed and accuracy of recognition. However, these deep learning methods can be effected by visual disturbances, such as lightning conditions, skin texture and other user-specific features. Data-driven approaches could reduce the performance degradation caused by these visual disturbances using models pretrained on large-scale datasets. But these methods often require large amounts of training data and computational resources, making them costly. To reduce the influence of user-specific features and enhance performance with limited data, this paper proposed a landmark guided visual feature extractor. Facial landmarks are used as auxiliary information to aid in training the visual feature extractor. A spatio-temporal multi-graph convolutional network is designed to fully exploit the spatial locations and spatio-temporal features of facial landmarks. Additionally, a multi-level lip dynamic fusion framework is introduced to combine the spatio-temporal features of the landmarks with the visual features extracted from the raw video frames. Experimental results show that this approach performs well with limited data and also improves the model's accuracy on unseen speakers.
Abstract (translated)
视觉语音识别是一种在无声视频中识别口语内容的技术，在近年来引起了广泛关注。数据驱动的深度学习方法的进步极大地提高了识别的速度和准确性。然而，这些深度学习方法可能会受到光照条件、皮肤纹理和其他用户特定特征等视觉干扰的影响。基于数据的方法可以通过使用大规模预训练数据集来减少这些问题导致的表现下降。但是，这种方法通常需要大量的训练数据和计算资源，成本较高。 为了减轻用户特有特征的影响，并在有限的数据下提高性能，本文提出了一种由面部地标引导的视觉特征提取器。面部地标被用作辅助信息，以帮助训练视觉特征提取器。设计了一个时空多图卷积网络来充分利用面部地标的时空位置和时空特征。此外，还引入了多层次唇部动态融合框架，将地标点的时空特性与从原始视频帧中提取出的视觉特征相结合。 实验结果表明，在有限的数据下该方法表现良好，并且提高了模型在未见过说话者上的准确性。
URL
https://arxiv.org/abs/2508.07233
