Abstract
Audio-driven talking head synthesis strives to generate lifelike video portraits from provided audio. The diffusion model, recognized for its superior quality and robust generalization, has been explored for this task. However, establishing a robust correspondence between temporal audio cues and corresponding spatial facial expressions with diffusion models remains a significant challenge in talking head generation. To bridge this gap, we present DreamHead, a hierarchical diffusion framework that learns spatial-temporal correspondences in talking head synthesis without compromising the model's intrinsic quality and adaptability.~DreamHead learns to predict dense facial landmarks from audios as intermediate signals to model the spatial and temporal correspondences.~Specifically, a first hierarchy of audio-to-landmark diffusion is first designed to predict temporally smooth and accurate landmark sequences given audio sequence signals. Then, a second hierarchy of landmark-to-image diffusion is further proposed to produce spatially consistent facial portrait videos, by modeling spatial correspondences between the dense facial landmark and appearance. Extensive experiments show that proposed DreamHead can effectively learn spatial-temporal consistency with the designed hierarchical diffusion and produce high-fidelity audio-driven talking head videos for multiple identities.
Abstract (translated)
音频驱动的讲话头合成旨在从提供的音频中生成逼真的视频肖像。扩散模型因其高质量和稳健的泛化能力而受到广泛探索。然而，在扩散模型中建立时空音频线索与相应面部表情的可靠对应仍然是一个重要的挑战。为了弥合这个差距，我们提出了DreamHead，一种分层扩散框架，它在不损害模型固有质量和适应性的前提下，从讲话头合成中学习时空对应关系。DreamHead从音频中预测丰富的面部关键点作为中间信号，以建模时空和面部表情的对应关系。具体来说，我们首先设计了一个音频到关键点的扩散层次结构，用于预测给定音频序列信号的时变平滑和准确的关键点序列。然后，我们进一步提出了一个地标到图像的扩散层次结构，通过建模密集面部关键点和外观之间的空间对应关系，产生具有空间一致性的面部肖像视频。大量实验证明，所提出的DreamHead可以有效地利用设计的分层扩散学习时空和面部表情的对应关系，并为多个身份生成高保真的音频驱动讲话头视频。
URL
https://arxiv.org/abs/2409.10281
