Abstract
Portrait Fidelity Generation is a prominent research area in generative models, with a primary focus on enhancing both controllability and fidelity. Current methods face challenges in generating high-fidelity portrait results when faces occupy a small portion of the image with a low resolution, especially in multi-person group photo settings. To tackle these issues, we propose a systematic solution called MagicID, based on a self-constructed million-level multi-modal dataset named IDZoom. MagicID consists of Multi-Mode Fusion training strategy (MMF) and DDIM Inversion based ID Restoration inference framework (DIIR). During training, MMF iteratively uses the skeleton and landmark modalities from IDZoom as conditional guidance. By introducing the Clone Face Tuning in training stage and Mask Guided Multi-ID Cross Attention (MGMICA) in inference stage, explicit constraints on face positional features are achieved for multi-ID group photo generation. The DIIR aims to address the issue of artifacts. The DDIM Inversion is used in conjunction with face landmarks, global and local face features to achieve face restoration while keeping the background unchanged. Additionally, DIIR is plug-and-play and can be applied to any diffusion-based portrait generation method. To validate the effectiveness of MagicID, we conducted extensive comparative and ablation experiments. The experimental results demonstrate that MagicID has significant advantages in both subjective and objective metrics, and achieves controllable generation in multi-person scenarios.
Abstract (translated)
肖像可靠性生成是一个在生成模型领域突出研究领域的知名领域，主要关注提高可控制性和可靠性。当面对低分辨率时，当前方法在生成高保真度肖像结果时存在挑战，尤其是在多人物群照设置中。为解决这些问题，我们提出了一个基于自构建的100万级多模态数据集IDZoom的系统解决方案，称为MagicID。MagicID包括多模态融合训练策略（MMF）和基于ID修复的DDIM Inversion。在训练过程中，MMF使用IDZoom中的骨架和关键点模态作为条件指导。通过引入训练阶段的人脸克隆和引导多ID照Cross注意力（MGMICA）以及在推理阶段使用DDIM Inversion，实现了对面部位置特征的显式约束，用于多ID照群照生成。 DIIR旨在解决伪影问题。将DDIM Inversion与面部关键点、全局和局部面部特征相结合，可以在保留背景不变的情况下实现面部修复。此外，DIIR是可插拔的，可以应用于任何基于扩散的肖像生成方法。为了验证MagicID的有效性，我们进行了广泛的比较和消融实验。实验结果表明，MagicID在主观和客观指标上具有显著优势，并在多人物群照场景中实现了可控制性的生成。
URL
https://arxiv.org/abs/2408.09248
