Abstract
Emotion is a critical component of artificial social intelligence. However, while current methods excel in lip synchronization and image quality, they often fail to generate accurate and controllable emotional expressions while preserving the subject's identity. To address this challenge, we introduce RealTalk, a novel framework for synthesizing emotional talking heads with high emotion accuracy, enhanced emotion controllability, and robust identity preservation. RealTalk employs a variational autoencoder (VAE) to generate 3D facial landmarks from driving audio, which are concatenated with emotion-label embeddings using a ResNet-based landmark deformation model (LDM) to produce emotional landmarks. These landmarks and facial blendshape coefficients jointly condition a novel tri-plane attention Neural Radiance Field (NeRF) to synthesize highly realistic emotional talking heads. Extensive experiments demonstrate that RealTalk outperforms existing methods in emotion accuracy, controllability, and identity preservation, advancing the development of socially intelligent AI systems.
Abstract (translated)
情绪是人工社会智能的一个关键组成部分。然而，当前的方法虽然在唇部同步和图像质量方面表现出色，但在生成准确且可控的情绪表达同时保持主体身份方面往往存在不足。为了解决这一挑战，我们引入了RealTalk，这是一种新型框架，用于合成高情感准确性、增强的情感控制能力和稳健的身份保留的对话头。 RealTalk 使用变分自动编码器（VAE）从驱动音频中生成3D面部标志点，并使用基于ResNet的面部标志变形模型（LDM）将这些标志点与情绪标签嵌入物连接起来，以产生具有情感的表情。此外，这些表情和面部混合形状系数共同作用于一种新的三平面注意力神经辐射场（NeRF），用于合成高度逼真的带有情感的对话头。 广泛的实验表明，RealTalk 在情感准确性、控制能力和身份保留方面优于现有方法，从而推动了社会智能AI系统的发展。
URL
https://arxiv.org/abs/2508.12163
