Abstract
Precise audio-visual synchronization in speech videos is crucial for content quality and viewer comprehension. Existing methods have made significant strides in addressing this challenge through rule-based approaches and end-to-end learning techniques. However, these methods often rely on limited audio-visual representations and suboptimal learning strategies, potentially constraining their effectiveness in more complex scenarios. To address these limitations, we present UniSync, a novel approach for evaluating audio-visual synchronization using embedding similarities. UniSync offers broad compatibility with various audio representations (e.g., Mel spectrograms, HuBERT) and visual representations (e.g., RGB images, face parsing maps, facial landmarks, 3DMM), effectively handling their significant dimensional differences. We enhance the contrastive learning framework with a margin-based loss component and cross-speaker unsynchronized pairs, improving discriminative capabilities. UniSync outperforms existing methods on standard datasets and demonstrates versatility across diverse audio-visual representations. Its integration into talking face generation frameworks enhances synchronization quality in both natural and AI-generated content.
Abstract (translated)
在语音视频中，精确的音视频同步对于内容质量和观众理解至关重要。现有方法通过基于规则的方法和端到端学习技术在解决这一挑战方面取得了重大进展。然而，这些方法往往依赖于有限的音频-视觉表示形式以及次优的学习策略，在更复杂的场景下可能限制了它们的效果。为了解决这些问题，我们提出了UniSync，这是一种使用嵌入相似性来评估音视频同步的新方法。 UniSync具有广泛的兼容性，能够处理各种音频表示（例如梅尔光谱图、HuBERT）和视觉表示（如RGB图像、面部解析地图、面部特征点、3DMM），并且能够有效应对它们之间显著的维度差异。我们通过加入基于余量的损失组件以及跨说话人的不同步对来增强对比学习框架，从而提高其判别能力。 在标准数据集上，UniSync的表现优于现有方法，并且在各种音频-视觉表示形式中展现了灵活性。将其集成到面部生成框架中可以提升自然内容和人工智能生成内容中的同步质量。
URL
https://arxiv.org/abs/2503.16357
