Abstract
While state-of-the-art audio-video generation models like Veo3 and Sora2 demonstrate remarkable capabilities, their closed-source nature makes their architectures and training paradigms inaccessible. To bridge this gap in accessibility and performance, we introduce UniTalking, a unified, end-to-end diffusion framework for generating high-fidelity speech and lip-synchronized video. At its core, our framework employs Multi-Modal Transformer Blocks to explicitly model the fine-grained temporal correspondence between audio and video latent tokens via a shared self-attention mechanism. By leveraging powerful priors from a pre-trained video generation model, our framework ensures state-of-the-art visual fidelity while enabling efficient training. Furthermore, UniTalking incorporates a personalized voice cloning capability, allowing the generation of speech in a target style from a brief audio reference. Qualitative and quantitative results demonstrate that our method produces highly realistic talking portraits, achieving superior performance over existing open-source approaches in lip-sync accuracy, audio naturalness, and overall perceptual quality.
Abstract (translated)
虽然像Veo3和Sora2这样的最先进的音频视频生成模型展示了非凡的能力，但它们的闭源性质使其架构和训练范式对外界不可访问。为了弥合这一可访问性和性能差距，我们引入了UniTalking，这是一种统一的端到端扩散框架，用于生成高保真语音和唇部同步视频。我们的框架的核心采用了多模态Transformer模块，通过共享自注意力机制明确地建模音频和视频潜在标记之间的细粒度时间对应关系。 借助预训练视频生成模型的强大先验知识，我们的框架确保了视觉保真的领先水平，并实现了高效的训练过程。此外，UniTalking还集成了个性化的语音克隆功能，可以从简短的音频参考中生成目标风格下的语音。 定性和定量的结果表明，我们所提出的方法能够生成高度逼真的话面图像，在唇同步准确性、音频自然度和整体感知质量方面优于现有的开源方法。
URL
https://arxiv.org/abs/2603.01418
