Abstract
We propose a two-stage framework for audio-driven talking head generation with fine-grained expression control via facial Action Units (AUs). Unlike prior methods relying on emotion labels or implicit AU conditioning, our model explicitly maps AUs to 2D facial landmarks, enabling physically grounded, per-frame expression control. In the first stage, a variational motion generator predicts temporally coherent landmark sequences from audio and AU intensities. In the second stage, a diffusion-based synthesizer generates realistic, lip-synced videos conditioned on these landmarks and a reference image. This separation of motion and appearance improves expression accuracy, temporal stability, and visual realism. Experiments on the MEAD dataset show that our method outperforms state-of-the-art baselines across multiple metrics, demonstrating the effectiveness of explicit AU-to-landmark modeling for expressive talking head generation.
Abstract (translated)
我们提出了一种基于音频驱动的面部表情精细控制的说话头部生成两阶段框架，该框架利用面部动作单元（Action Units，AUs）。与以往依赖情感标签或隐式AU条件的方法不同，我们的模型明确地将AU映射到2D面部特征点上，从而使表情控制更加物理现实，并且可以逐帧进行。在第一阶段中，一个变分运动生成器根据音频和AU强度预测出时间连贯的特征点序列；第二阶段则是一个基于扩散的合成器，该合成器根据这些特征点以及参考图像生成逼真的口型同步视频。这种将动作与外观分离的方法提高了表情准确性、时间稳定性及视觉真实感。 在MEAD数据集上的实验表明，我们提出的方法在多个指标上优于当前最先进的基准方法，证明了显式AU到特征点建模在表达性说话头部生成中的有效性。
URL
https://arxiv.org/abs/2509.19749
