Abstract
Despite the similar structures of human faces, existing face alignment methods cannot learn unified knowledge from multiple datasets with different landmark annotations. The limited training samples in a single dataset commonly result in fragile robustness in this field. To mitigate knowledge discrepancies among different datasets and train a task-agnostic unified face alignment (TUFA) framework, this paper presents a strategy to unify knowledge from multiple datasets. Specifically, we calculate a mean face shape for each dataset. To explicitly align these mean shapes on an interpretable plane based on their semantics, each shape is then incorporated with a group of semantic alignment embeddings. The 2D coordinates of these aligned shapes can be viewed as the anchors of the plane. By encoding them into structure prompts and further regressing the corresponding facial landmarks using image features, a mapping from the plane to the target faces is finally established, which unifies the learning target of different datasets. Consequently, multiple datasets can be utilized to boost the generalization ability of the model. The successful mitigation of discrepancies also enhances the efficiency of knowledge transferring to a novel dataset, significantly boosts the performance of few-shot face alignment. Additionally, the interpretable plane endows TUFA with a task-agnostic characteristic, enabling it to locate landmarks unseen during training in a zero-shot manner. Extensive experiments are carried on seven benchmarks and the results demonstrate an impressive improvement in face alignment brought by knowledge discrepancies mitigation.
Abstract (translated)
尽管人类面部结构相似，现有的面部对齐方法无法从具有不同地标标注的多个数据集中学习统一的知识。单一数据集中的有限训练样本通常会导致该领域的脆弱性增强。为了减轻不同数据集之间的知识差异，并培训一个任务无关的统一面部对齐（TUFA）框架，本文提出了一种策略，以将来自多个数据集的知识进行统一化。具体来说，我们为每个数据集计算平均面部形状。为了让这些平均形状基于其语义在可解释平面上显式对齐，接着给每一种形状加入一组语义对齐嵌入。这些对齐后的形状的二维坐标可以被视为该平面的锚点。通过将它们编码成结构提示，并进一步使用图像特征回归对应的面部地标，最终建立了一个从平面到目标面部的映射关系，从而统一了不同数据集的学习目标。因此，多个数据集可以被用来增强模型的泛化能力。成功地缓解了差异性也提高了知识向新数据集转移的效率，显著提升了少样本面部对齐的表现。此外，可解释的平面赋予TUFA任务无关特性，使其能够以零样本的方式定位训练中未见过的地标。在七个基准上的广泛实验表明，通过减少知识差异所带来的人脸对齐性能有了显著改进。
URL
https://arxiv.org/abs/2503.22359
