Abstract
While diffusion models have shown great potential in portrait generation, generating expressive, coherent, and controllable cinematic portrait videos remains a significant challenge. Existing intermediate signals for portrait generation, such as 2D landmarks and parametric models, have limited disentanglement capabilities and cannot express personalized details due to their sparse or low-rank representation. Therefore, existing methods based on these models struggle to accurately preserve subject identity and expressions, hindering the generation of highly expressive portrait videos. To overcome these limitations, we propose a high-fidelity personalized head representation that more effectively disentangles expression and identity. This representation captures both static, subject-specific global geometry and dynamic, expression-related details. Furthermore, we introduce an expression transfer module to achieve personalized transfer of head pose and expression details between different identities. We use this sophisticated and highly expressive head model as a conditional signal to train a diffusion transformer (DiT)-based generator to synthesize richly detailed portrait videos. Extensive experiments on self- and cross-reenactment tasks demonstrate that our method outperforms previous models in terms of identity preservation, expression accuracy, and temporal stability, particularly in capturing fine-grained details of complex motion.
Abstract (translated)
尽管扩散模型在肖像生成方面展现出了巨大的潜力，但生成具有表现力、连贯性和可控性的电影级肖像视频仍然是一项重大挑战。现有用于肖像生成的中间信号（如2D地标和参数化模型）由于其稀疏或低秩表示，解耦能力有限，无法表达个性化的细节。因此，基于这些模型的方法难以准确地保持人物的身份和表情，阻碍了高度表现力的肖像视频的生成。 为克服这些限制，我们提出了一种高保真度的个性化头部表征方法，这种方法能够更有效地分离表情和身份。该表征捕捉到了静态的、个性化的全局几何特征以及动态的表情相关细节。此外，我们还引入了一个表情迁移模块，用于实现不同人物之间头部姿态和表情细节的个性化转移。 我们将这一复杂且高度表现力的头部模型作为条件信号来训练基于扩散变换器（DiT）的生成器，以合成具有丰富细节的肖像视频。在自我再现和跨身份再现任务中的广泛实验表明，我们的方法在身份保持、表情准确性以及时间稳定性方面均优于先前的方法，特别是在捕捉复杂动作的细粒度细节方面尤为突出。
URL
https://arxiv.org/abs/2602.19900
