Abstract
Video-driven 3D facial animation transfer aims to drive avatars to reproduce the expressions of actors. Existing methods have achieved remarkable results by constraining both geometric and perceptual consistency. However, geometric constraints (like those designed on facial landmarks) are insufficient to capture subtle emotions, while expression features trained on classification tasks lack fine granularity for complex emotions. To address this, we propose \textbf{FreeAvatar}, a robust facial animation transfer method that relies solely on our learned expression representation. Specifically, FreeAvatar consists of two main components: the expression foundation model and the facial animation transfer model. In the first component, we initially construct a facial feature space through a face reconstruction task and then optimize the expression feature space by exploring the similarities among different expressions. Benefiting from training on the amounts of unlabeled facial images and re-collected expression comparison dataset, our model adapts freely and effectively to any in-the-wild input facial images. In the facial animation transfer component, we propose a novel Expression-driven Multi-avatar Animator, which first maps expressive semantics to the facial control parameters of 3D avatars and then imposes perceptual constraints between the input and output images to maintain expression consistency. To make the entire process differentiable, we employ a trained neural renderer to translate rig parameters into corresponding images. Furthermore, unlike previous methods that require separate decoders for each avatar, we propose a dynamic identity injection module that allows for the joint training of multiple avatars within a single network.
Abstract (translated)
视频驱动的3D面部动画传输旨在将虚拟角色复制演员的表情。现有的方法通过约束几何和感知一致性取得了显著的成果。然而，约束面部特征点的几何限制（如设计在面部特征点上的限制）不足以捕捉细微的情感，而经过分类任务训练的表情特征缺乏复杂情感的细粒度。为了解决这个问题，我们提出了FreeAvatar，一种仅依赖我们所学到的表情表示的稳健面部动画传输方法。具体来说，FreeAvatar包括两个主要组件：表情基础模型和面部动画传输模型。在第一个组件中，我们通过面部重建任务首先构建了一个面部特征空间，然后通过探索不同表情之间的相似性来优化表情特征空间。得益于在未标注面部图像的训练和重新收集表情比较数据集上训练，我们的模型可以自由地适应任何野外输入面部图像。在面部动画传输组件中，我们提出了一个新颖的表达驱动多虚拟角色动画器，它首先将表情语义映射到3D虚拟角色的面部控制参数上，然后对输入和输出图像之间的感知约束进行约束，以保持表情一致性。为了使整个过程具有可导性，我们使用训练好的神经渲染器将 rig 参数转换为相应的图像。此外，与之前的方法不同，我们提出了一个动态身份注入模块，允许在单个网络中联合训练多个虚拟角色。
URL
https://arxiv.org/abs/2409.13180
