Abstract
Audio-driven video portrait synthesis is a crucial and useful technology in virtual human interaction and film-making applications. Recent advancements have focused on improving the image fidelity and lip-synchronization. However, generating accurate emotional expressions is an important aspect of realistic talking-head generation, which has remained underexplored in previous works. We present a novel system in this paper for synthesizing high-fidelity, audio-driven video portraits with accurate emotional expressions. Specifically, we utilize a variational autoencoder (VAE)-based audio-to-motion module to generate facial landmarks. These landmarks are concatenated with emotional embeddings to produce emotional landmarks through our motion-to-emotion module. These emotional landmarks are then used to render realistic emotional talking-head video using a Neural Radiance Fields (NeRF)-based emotion-to-video module. Additionally, we propose a pose sampling method that generates natural idle-state (non-speaking) videos in response to silent audio inputs. Extensive experiments demonstrate that our method obtains more accurate emotion generation with higher fidelity.
Abstract (translated)
音频驱动的视频肖像合成技术在虚拟人互动和电影制作应用中是一项关键且有用的技术。最近的研究进展主要集中在提高图像保真度和唇部同步上。然而，生成准确的情感表达是实现逼真的说话头像生成的重要方面，在以往的工作中这一领域尚未得到充分探索。本文提出了一种新颖的系统，用于合成具有高保真度和准确情感表达的音频驱动视频肖像。具体而言，我们使用基于变分自编码器（VAE）的音频到动作模块来生成面部特征点。这些特征点与情感嵌入结合通过我们的运动到情感模块产生情感特征点。随后，这些情感特征点被用于利用基于神经辐射场（NeRF）的情感到视频模块渲染逼真的带有情感的说话头像视频。此外，我们提出了一种姿态采样方法，该方法能够根据静音音频输入生成自然的空闲状态（非讲话）视频。广泛的实验表明，我们的方法在更准确地生成情感方面获得了更高的保真度。
URL
https://arxiv.org/abs/2410.17262
