Abstract
Landmark-guided character animation generation is an important field. Generating character animations with facial features consistent with a reference image remains a significant challenge in conditional video generation, especially involving complex motions like dancing. Existing methods often fail to maintain facial feature consistency due to mismatches between the facial landmarks extracted from source videos and the target facial features in the reference image. To address this problem, we propose a facial landmark transformation method based on the 3D Morphable Model (3DMM). We obtain transformed landmarks that align with the target facial features by reconstructing 3D faces from the source landmarks and adjusting the 3DMM parameters to match the reference image. Our method improves the facial consistency between the generated videos and the reference images, effectively improving the facial feature mismatch problem.
Abstract (translated)
地标引导的角色动画生成是一个重要的领域。在条件视频生成中，特别是涉及复杂动作如舞蹈时，生成与参考图像面部特征一致的角色动画仍是一项重大挑战。现有方法往往因源视频提取的面部地标与参考图像中的目标面部特征不匹配而无法保持面部特征的一致性。为解决这一问题，我们提出了一种基于3D可变形模型（3DMM）的面部地标变换方法。通过从源地标重建3D人脸，并调整3DMM参数以匹配参考图像，我们获得了与目标面部特征对齐的变换后的地标。我们的方法提高了生成视频与参考图像之间的面部一致性，有效解决了面部特征不匹配的问题。
URL
https://arxiv.org/abs/2412.08976
