Facial_Landmark
-
Cascaded Dual Vision Transformer for Accurate Facial Landmark Detection
2024-11-08 07:26:39Ziqiang Dang, Jianfang Li, Lin LiuAbstract
Facial landmark detection is a fundamental problem in computer vision for many downstream applications. This paper introduces a new facial landmark detector based on vision transformers, which consists of two unique designs: Dual Vision Transformer (D-ViT) and Long Skip Connections (LSC). Based on the observation that the channel dimension of feature maps essentially represents the linear bases of the heatmap space, we propose learning the interconnections between these linear bases to model the inherent geometric relations among landmarks via Channel-split ViT. We integrate such channel-split ViT into the standard vision transformer (i.e., spatial-split ViT), forming our Dual Vision Transformer to constitute the prediction blocks. We also suggest using long skip connections to deliver low-level image features to all prediction blocks, thereby preventing useful information from being discarded by intermediate supervision. Extensive experiments are conducted to evaluate the performance of our proposal on the widely used benchmarks, i.e., WFLW, COFW, and 300W, demonstrating that our model outperforms the previous SOTAs across all three benchmarks.
Abstract (translated)
面部特征点检测是计算机视觉中许多下游应用的基础问题。本文介绍了一种基于视觉变换器的新面部特征点检测器，该检测器包含两个独特的设计：双视觉变换器（D-ViT）和长跳跃连接（LSC）。根据特征图的通道维度实际上表示热图空间线性基底这一观察结果，我们提出了通过通道分割ViT学习这些线性基底之间的相互联系，以建模地标间的固有几何关系。我们将这种通道分割ViT整合到标准视觉变换器（即空间分割ViT）中，形成我们的双视觉变换器，构成预测块。此外，我们还建议使用长跳跃连接将低级图像特征传递给所有预测块，从而防止因中间监督而丢失有用信息。我们在广泛使用的基准测试集上进行了广泛的实验，包括WFLW、COFW和300W，结果表明我们的模型在这三个基准测试集中都优于之前的最佳技术水平。
URL
https://arxiv.org/abs/2411.07167
PDF
-
Face Anonymization Made Simple
2024-11-01 17:45:21Han-Wei Kung, Tuomas Varanka, Sanjay Saha, Terence Sim, Nicu SebeAbstract
Current face anonymization techniques often depend on identity loss calculated by face recognition models, which can be inaccurate and unreliable. Additionally, many methods require supplementary data such as facial landmarks and masks to guide the synthesis process. In contrast, our approach uses diffusion models with only a reconstruction loss, eliminating the need for facial landmarks or masks while still producing images with intricate, fine-grained details. We validated our results on two public benchmarks through both quantitative and qualitative evaluations. Our model achieves state-of-the-art performance in three key areas: identity anonymization, facial attribute preservation, and image quality. Beyond its primary function of anonymization, our model can also perform face swapping tasks by incorporating an additional facial image as input, demonstrating its versatility and potential for diverse applications. Our code and models are available at this https URL .
Abstract (translated)
当前的面部匿名化技术通常依赖于由人脸识别模型计算出的身份损失，这可能会导致不准确和不可靠的结果。此外，许多方法需要额外的数据，如面部特征点和遮罩来指导合成过程。相比之下，我们的方法仅使用扩散模型和重建损失，不需要面部特征点或遮罩，同时仍能生成具有复杂精细细节的图像。我们在两个公开基准上通过定量和定性评估验证了结果。我们的模型在三个关键领域达到了最先进的性能：身份匿名化、面部属性保留和图像质量。除了主要的匿名功能外，通过将额外的面部图像作为输入，我们的模型还可以执行面部交换任务，展示了其多样性和潜在的应用前景。我们的代码和模型可以在以下链接中获取: [此 HTTPS URL]。
URL
https://arxiv.org/abs/2411.00762
PDF
-
AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels
2024-10-29 19:57:10Ali Pourramezan Fard, Mohammad Mehdi Hosseini, Timothy D. Sweeny, Mohammad H. MahoorAbstract
Automated Facial Expression Recognition (FER) is challenging due to intra-class variations and inter-class similarities. FER can be especially difficult when facial expressions reflect a mixture of various emotions (aka compound expressions). Existing FER datasets, such as AffectNet, provide discrete emotion labels (hard-labels), where a single category of emotion is assigned to an expression. To alleviate inter- and intra-class challenges, as well as provide a better facial expression descriptor, we propose a new approach to create FER datasets through a labeling method in which an image is labeled with more than one emotion (called soft-labels), each with different confidences. Specifically, we introduce the notion of soft-labels for facial expression datasets, a new approach to affective computing for more realistic recognition of facial expressions. To achieve this goal, we propose a novel methodology to accurately calculate soft-labels: a vector representing the extent to which multiple categories of emotion are simultaneously present within a single facial expression. Finding smoother decision boundaries, enabling multi-labeling, and mitigating bias and imbalanced data are some of the advantages of our proposed method. Building upon AffectNet, we introduce AffectNet+, the next-generation facial expression dataset. This dataset contains soft-labels, three categories of data complexity subsets, and additional metadata such as age, gender, ethnicity, head pose, facial landmarks, valence, and arousal. AffectNet+ will be made publicly accessible to researchers.
Abstract (translated)
自动化面部表情识别（FER）具有挑战性，主要由于类内变化和类间相似性的存在。当面部表情反映多种情感的混合时（即复合表达），FER尤其困难。现有的FER数据集，如AffectNet，提供离散的情感标签（硬标签），其中一种情绪类别被分配给一个表情。为了缓解类内和类间的挑战，并提供更好的面部表情描述符，我们提出了一种新的方法来创建FER数据集：通过标注方法对一张图片进行多于一种情感的标注（称为软标签），每种情感带有不同的置信度。具体来说，我们引入了面部表情数据集中软标签的概念，这是一种更为现实的情感计算新方法，用于面部表情识别。为了实现这一目标，我们提出了一种新的方法来准确计算软标签：一个向量表示在单个面部表情中同时存在多种情绪的程度。我们的提议方法具有发现平滑决策边界、支持多标注以及减轻偏差和数据不平衡的优势。基于AffectNet，我们引入了下一代面部表情数据集AffectNet+。该数据集包含软标签、三个类别的数据复杂度子集以及额外的元数据如年龄、性别、种族、头部姿态、面部特征点、效价和唤醒水平。AffectNet+将向研究人员公开提供访问权限。
URL
https://arxiv.org/abs/2410.22506
PDF
-
R2I-rPPG: A Robust Region of Interest Selection Method for Remote Photoplethysmography to Extract Heart Rate
2024-10-21 10:27:57Sandeep Nagar, Mark Hasegawa-Johnson, David G. Beiser, Narendra AhujaAbstract
The COVID-19 pandemic has underscored the need for low-cost, scalable approaches to measuring contactless vital signs, either during initial triage at a healthcare facility or virtual telemedicine visits. Remote photoplethysmography (rPPG) can accurately estimate heart rate (HR) when applied to close-up videos of healthy volunteers in well-lit laboratory settings. However, results from such highly optimized laboratory studies may not be readily translated to healthcare settings. One significant barrier to the practical application of rPPG in health care is the accurate localization of the region of interest (ROI). Clinical or telemedicine visits may involve sub-optimal lighting, movement artifacts, variable camera angle, and subject distance. This paper presents an rPPG ROI selection method based on 3D facial landmarks and patient head yaw angle. We then demonstrate the robustness of this ROI selection method when coupled to the Plane-Orthogonal-to-Skin (POS) rPPG method when applied to videos of patients presenting to an Emergency Department for respiratory complaints. Our results demonstrate the effectiveness of our proposed approach in improving the accuracy and robustness of rPPG in a challenging clinical environment.
Abstract (translated)
URL
https://arxiv.org/abs/2410.15851
PDF
-
POPoS: Improving Efficient and Robust Facial Landmark Detection with Parallel Optimal Position Search
2024-10-12 16:28:40Chong-Yang Xiang, Jun-Yan He, Zhi-Qi Cheng, Xiao Wu, Xian-Sheng HuaAbstract
Achieving a balance between accuracy and efficiency is a critical challenge in facial landmark detection (FLD). This paper introduces the Parallel Optimal Position Search (POPoS), a high-precision encoding-decoding framework designed to address the fundamental limitations of traditional FLD methods. POPoS employs three key innovations: (1) Pseudo-range multilateration is utilized to correct heatmap errors, enhancing the precision of landmark localization. By integrating multiple anchor points, this approach minimizes the impact of individual heatmap inaccuracies, leading to robust overall positioning. (2) To improve the pseudo-range accuracy of selected anchor points, a new loss function, named multilateration anchor loss, is proposed. This loss function effectively enhances the accuracy of the distance map, mitigates the risk of local optima, and ensures optimal solutions. (3) A single-step parallel computation algorithm is introduced, significantly enhancing computational efficiency and reducing processing time. Comprehensive evaluations across five benchmark datasets demonstrate that POPoS consistently outperforms existing methods, particularly excelling in low-resolution scenarios with minimal computational overhead. These features establish POPoS as a highly efficient and accurate tool for FLD, with broad applicability in real-world scenarios. The code is available at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2410.09583
PDF
-
3D2M Dataset: A 3-Dimension diverse Mesh Dataset
2024-10-09 20:29:16Sankarshan DasguptaAbstract
Three-dimensional (3D) reconstruction has emerged as a prominent area of research, attracting significant attention from academia and industry alike. Among the various applications of 3D reconstruction, facial reconstruction poses some of the most formidable challenges. Additionally, each individuals facial structure is unique, requiring algorithms to be robust enough to handle this variability while maintaining fidelity to the original features. This article presents a comprehensive dataset of 3D meshes featuring a diverse range of facial structures and corresponding facial landmarks. The dataset comprises 188 3D facial meshes, including 73 from female candidates and 114 from male candidates. It encompasses a broad representation of ethnic backgrounds, with contributions from 45 different ethnicities, ensuring a rich diversity in facial characteristics. Each facial mesh is accompanied by key points that accurately annotate the relevant features, facilitating precise analysis and manipulation. This dataset is particularly valuable for applications such as facial re targeting, the study of facial structure components, and real-time person representation in video streams. By providing a robust resource for researchers and developers, it aims to advance the field of 3D facial reconstruction and related technologies.
Abstract (translated)
URL
https://arxiv.org/abs/2410.07415
PDF
-
Toward Scalable Image Feature Compression: A Content-Adaptive and Diffusion-Based Approach
2024-10-08 15:48:34Sha Guo, Zhuo Chen, Yang Zhao, Ning Zhang, Xiaotong Li, Lingyu DuanAbstract
Traditional image codecs emphasize signal fidelity and human perception, often at the expense of machine vision tasks. Deep learning methods have demonstrated promising coding performance by utilizing rich semantic embeddings optimized for both human and machine vision. However, these compact embeddings struggle to capture fine details such as contours and textures, resulting in imperfect reconstructions. Furthermore, existing learning-based codecs lack scalability. To address these limitations, this paper introduces a content-adaptive diffusion model for scalable image compression. The proposed method encodes fine textures through a diffusion process, enhancing perceptual quality while preserving essential features for machine vision tasks. The approach employs a Markov palette diffusion model combined with widely used feature extractors and image generators, enabling efficient data compression. By leveraging collaborative texture-semantic feature extraction and pseudo-label generation, the method accurately captures texture information. A content-adaptive Markov palette diffusion model is then applied to represent both low-level textures and high-level semantic content in a scalable manner. This framework offers flexible control over compression ratios by selecting intermediate diffusion states, eliminating the need for retraining deep learning models at different operating points. Extensive experiments demonstrate the effectiveness of the proposed framework in both image reconstruction and downstream machine vision tasks such as object detection, segmentation, and facial landmark detection, achieving superior perceptual quality compared to state-of-the-art methods.
Abstract (translated)
URL
https://arxiv.org/abs/2410.06149
PDF
-
Audio-Driven Emotional 3D Talking-Head Generation
2024-10-07 08:23:05Wenqing Wang, Yun FuAbstract
Audio-driven video portrait synthesis is a crucial and useful technology in virtual human interaction and film-making applications. Recent advancements have focused on improving the image fidelity and lip-synchronization. However, generating accurate emotional expressions is an important aspect of realistic talking-head generation, which has remained underexplored in previous works. We present a novel system in this paper for synthesizing high-fidelity, audio-driven video portraits with accurate emotional expressions. Specifically, we utilize a variational autoencoder (VAE)-based audio-to-motion module to generate facial landmarks. These landmarks are concatenated with emotional embeddings to produce emotional landmarks through our motion-to-emotion module. These emotional landmarks are then used to render realistic emotional talking-head video using a Neural Radiance Fields (NeRF)-based emotion-to-video module. Additionally, we propose a pose sampling method that generates natural idle-state (non-speaking) videos in response to silent audio inputs. Extensive experiments demonstrate that our method obtains more accurate emotion generation with higher fidelity.
Abstract (translated)
音频驱动的视频肖像合成技术在虚拟人互动和电影制作应用中是一项关键且有用的技术。最近的研究进展主要集中在提高图像保真度和唇部同步上。然而，生成准确的情感表达是实现逼真的说话头像生成的重要方面，在以往的工作中这一领域尚未得到充分探索。本文提出了一种新颖的系统，用于合成具有高保真度和准确情感表达的音频驱动视频肖像。具体而言，我们使用基于变分自编码器（VAE）的音频到动作模块来生成面部特征点。这些特征点与情感嵌入结合通过我们的运动到情感模块产生情感特征点。随后，这些情感特征点被用于利用基于神经辐射场（NeRF）的情感到视频模块渲染逼真的带有情感的说话头像视频。此外，我们提出了一种姿态采样方法，该方法能够根据静音音频输入生成自然的空闲状态（非讲话）视频。广泛的实验表明，我们的方法在更准确地生成情感方面获得了更高的保真度。
URL
https://arxiv.org/abs/2410.17262
PDF
-
Visual Prompting in LLMs for Enhancing Emotion Recognition
2024-10-03 06:33:43Qixuan Zhang, Zhifeng Wang, Dylan Zhang, Wenjia Niu, Sabrina Caldwell, Tom Gedeon, Yang Liu, Zhenyue QinAbstract
Vision Large Language Models (VLLMs) are transforming the intersection of computer vision and natural language processing. Nonetheless, the potential of using visual prompts for emotion recognition in these models remains largely unexplored and untapped. Traditional methods in VLLMs struggle with spatial localization and often discard valuable global context. To address this problem, we propose a Set-of-Vision prompting (SoV) approach that enhances zero-shot emotion recognition by using spatial information, such as bounding boxes and facial landmarks, to mark targets precisely. SoV improves accuracy in face count and emotion categorization while preserving the enriched image context. Through a battery of experimentation and analysis of recent commercial or open-source VLLMs, we evaluate the SoV model's ability to comprehend facial expressions in natural environments. Our findings demonstrate the effectiveness of integrating spatial visual prompts into VLLMs for improving emotion recognition performance.
Abstract (translated)
视觉大型语言模型（VLLMs）正在改变计算机视觉和自然语言处理之间的交叉。然而，在這些模型中使用視覺提示進行情感識別的潛力仍然被大大開發和利用。傳統的VLLM方法在空間定位方面存在困難，通常會舍棄有價值的全身上下文。為了解決這個問題，我們提出了一个Set-of-Vision prompting（SoV）方法，通過使用空間信息（如邊界框和面部 landmarks）來精確標記目標，來增強 zero-shot情感識別的準確度。SoV在face count和 emotion categorization方面改善了準確度，同時保留了豐富的情感上下文。通過對最新商業或開源VLLM進行了大量實驗和分析，我們評估了SoV模型在自然環境中理解面部表情的能力。我們的研究結果表明，將空間視覺提示集成到VLLM中可以提高情感識別性能。
URL
https://arxiv.org/abs/2410.02244
PDF
-
Spatial Action Unit Cues for Interpretable Deep Facial Expression Recognition
2024-10-01 10:42:55Soufiane Belharbi, Marco Pedersoli, Alessandro Lameiras Koerich, Simon Bacon, Eric GrangerAbstract
Although state-of-the-art classifiers for facial expression recognition (FER) can achieve a high level of accuracy, they lack interpretability, an important feature for end-users. Experts typically associate spatial action units (AUs) from a codebook to facial regions for the visual interpretation of expressions. In this paper, the same expert steps are followed. A new learning strategy is proposed to explicitly incorporate AU cues into classifier training, allowing to train deep interpretable models. During training, this AU codebook is used, along with the input image expression label, and facial landmarks, to construct a AU heatmap that indicates the most discriminative image regions of interest w.r.t the facial expression. This valuable spatial cue is leveraged to train a deep interpretable classifier for FER. This is achieved by constraining the spatial layer features of a classifier to be correlated with AU heatmaps. Using a composite loss, the classifier is trained to correctly classify an image while yielding interpretable visual layer-wise attention correlated with AU maps, simulating the expert decision process. Our strategy only relies on image class expression for supervision, without additional manual annotations. Our new strategy is generic, and can be applied to any deep CNN- or transformer-based classifier without requiring any architectural change or significant additional training time. Our extensive evaluation on two public benchmarks RAF-DB, and AffectNet datasets shows that our proposed strategy can improve layer-wise interpretability without degrading classification performance. In addition, we explore a common type of interpretable classifiers that rely on class activation mapping (CAM) methods, and show that our approach can also improve CAM interpretability.
Abstract (translated)
尽管最先进的面部表情识别（FER）分类器可以实现高水平的准确性，但它们缺乏可解释性，这对于最终用户来说非常重要。专家通常将代码库中的空间动作单元（AUs）与表情进行视觉解释。在本文中，遵循了相同的专家步骤。提出了一种新的学习策略，将AU提示明确地融入分类器训练中，以训练深度可解释的模型。在训练过程中，使用了新的AU代码本和表情图像标签以及面部关键点来构建AU热力图，表示与面部表情最相关的图像区域。这个有价值的空间线索被用于训练FER的深度可解释分类器。通过约束分类器的空间层特征与AU热力图相关，实现了训练分类器正确分类图像，并产生与AU地图相关的可解释视觉层级的关注，模拟了专家的决策过程。我们的策略仅依赖于图像类别的表达进行监督，没有额外的手动注释。我们的新策略是通用的，可以应用于任何基于深度卷积神经网络（CNN）或Transformer的分类器，而不需要进行任何架构上的改变或额外的训练时间。我们对两个公开基准数据集RAF-DB和AffectNet进行的大量评估显示，与我们的建议的策略相比，可以在不降低分类性能的情况下提高层间可解释性。此外，我们还研究了依赖于分类激活映射（CAM）方法的常见可解释分类器，并证明了我们的方法也可以提高CAM的可解释性。
URL
https://arxiv.org/abs/2410.01848
PDF
-
VisioPhysioENet: Multimodal Engagement Detection using Visual and Physiological Signals
2024-09-24 14:36:19Alakhsimar Singh, Nischay Verma, Kanav Goyal, Amritpal Singh, Puneet Kumar, Xiaobai LiAbstract
This paper presents VisioPhysioENet, a novel multimodal system that leverages visual cues and physiological signals to detect learner engagement. It employs a two-level approach for visual feature extraction using the Dlib library for facial landmark extraction and the OpenCV library for further estimations. This is complemented by extracting physiological signals using the plane-orthogonal-to-skin method to assess cardiovascular activity. These features are integrated using advanced machine learning classifiers, enhancing the detection of various engagement levels. We rigorously evaluate VisioPhysioENet on the DAiSEE dataset, where it achieves an accuracy of 63.09%, demonstrating a superior ability to discern various levels of engagement compared to existing methodologies. The proposed system's code can be accessed at this https URL.
Abstract (translated)
本文介绍了一种名为VisioPhysioENet的多模态系统，该系统利用视觉提示和生理信号来检测学习者的参与度。它采用了一种双级方法，基于Dlib库进行面部关键点提取，并使用OpenCV库进行进一步估计。这通过使用平面与皮肤垂直的方法提取生理信号来评估心血管活动。这些特征使用先进的机器学习分类器进行整合，增强了各种参与度的检测。我们在DAiSEE数据集上对VisioPhysioENet进行了严格的评估，其精度为63.09%，表明其比现有方法更能够辨别各种参与度水平。该系统的代码可以在此链接访问：<https://www.academia.edu/39411041/VisioPhysioENet>
URL
https://arxiv.org/abs/2409.16126
PDF
-
Deep Learning Techniques for Automatic Lateral X-ray Cephalometric Landmark Detection: Is the Problem Solved?
2024-09-24 08:03:13Hongyuan Zhang, Ching-Wei Wang, Hikam Muzakky, Juan Dai, Xuguang Li, Chenglong Ma, Qian Wu, Xianan Cui, Kunlun Xu, Pengfei He, Dongqian Guo, Xianlong Wang, Hyunseok Lee, Zhangnan Zhong, Zhu Zhu, Bingsheng HuangAbstract
Localization of the craniofacial landmarks from lateral cephalograms is a fundamental task in cephalometric analysis. The automation of the corresponding tasks has thus been the subject of intense research over the past decades. In this paper, we introduce the "Cephalometric Landmark Detection (CL-Detection)" dataset, which is the largest publicly available and comprehensive dataset for cephalometric landmark detection. This multi-center and multi-vendor dataset includes 600 lateral X-ray images with 38 landmarks acquired with different equipment from three medical centers. The overarching objective of this paper is to measure how far state-of-the-art deep learning methods can go for cephalometric landmark detection. Following the 2023 MICCAI CL-Detection Challenge, we report the results of the top ten research groups using deep learning methods. Results show that the best methods closely approximate the expert analysis, achieving a mean detection rate of 75.719% and a mean radial error of 1.518 mm. While there is room for improvement, these findings undeniably open the door to highly accurate and fully automatic location of craniofacial landmarks. We also identify scenarios for which deep learning methods are still failing. Both the dataset and detailed results are publicly available online, while the platform will remain open for the community to benchmark future algorithm developments at this https URL.
Abstract (translated)
颅面标志从侧位X光片的本地化是一个关键的颅面测量分析任务。因此，在过去的几十年里，相应任务的自动化已成为研究的热点。在本文中，我们引入了“颅面标志检测（CL-Detection）”数据集，这是目前公开可用的最大且全面的颅面标志检测数据集。这个多中心、多供应商的数据集包括来自三个医疗中心的600个侧位X光图像，其中有38个标志物使用了不同的设备进行测量。本文的总体目标是测量最先进的深度学习方法在颅面标志检测方面的进展。在2023年MICCAI CL-Detection挑战中，我们报道了使用深度学习方法排名前十的研究组的成果。结果表明，最好的方法与专家分析的准确度非常接近，实现了平均检测率为75.719%和平均径向误差为1.518毫米。虽然仍有很多改进的空间，但这些发现无疑为准确且完全自动地定位颅面标志打开了大门。我们还指出了深度学习方法仍然失败的场景。该数据集和详细结果都可以公开获取，而平台将始终保持开放，以供社区在此处对未来的算法发展进行基准测试。
URL
https://arxiv.org/abs/2409.15834
PDF
-
QUB-PHEO: A Visual-Based Dyadic Multi-View Dataset for Intention Inference in Collaborative Assembly
2024-09-23 21:34:49Samuel Adebayo, Se\'an McLoone, Joost C. DessingAbstract
QUB-PHEO introduces a visual-based, dyadic dataset with the potential of advancing human-robot interaction (HRI) research in assembly operations and intention inference. This dataset captures rich multimodal interactions between two participants, one acting as a 'robot surrogate,' across a variety of assembly tasks that are further broken down into 36 distinct subtasks. With rich visual annotations, such as facial landmarks, gaze, hand movements, object localization, and more for 70 participants, QUB-PHEO offers two versions: full video data for 50 participants and visual cues for all 70. Designed to improve machine learning models for HRI, QUB-PHEO enables deeper analysis of subtle interaction cues and intentions, promising contributions to the field. The dataset will be available at this https URL subject to an End-User License Agreement (EULA).
Abstract (translated)
QUB-PHEO 是一个基于视觉的、二元数据集，具有推动人机交互（HRI）研究的发展以及装配操作和意图推理的潜力。这个数据集捕捉了两个参与者之间丰富的多模态互动，其中一个是“机器人代理”，在各种装配任务中进行操作。数据集进一步细分为 36 个不同的子任务。有了丰富的视觉注释，例如面部特征、眼神、手部动作、物体定位等，对于 70 名参与者，QUB-PHEO 提供了两种版本：对于 50 名参与者的完整视频数据，以及所有 70 名参与者的视觉提示。旨在提高 HRI 机器学习模型的性能，QUB-PHEO 使得对微妙的互动提示和意图进行更深刻的分析，为该领域做出贡献。数据集将在此链接上提供，受最终用户许可协议（EULA）约束。
URL
https://arxiv.org/abs/2409.15560
PDF
-
FreeAvatar: Robust 3D Facial Animation Transfer by Learning an Expression Foundation Model
2024-09-20 03:17:01Feng Qiu, Wei Zhang, Chen Liu, Rudong An, Lincheng Li, Yu Ding, Changjie Fan, Zhipeng Hu, Xin YuAbstract
Video-driven 3D facial animation transfer aims to drive avatars to reproduce the expressions of actors. Existing methods have achieved remarkable results by constraining both geometric and perceptual consistency. However, geometric constraints (like those designed on facial landmarks) are insufficient to capture subtle emotions, while expression features trained on classification tasks lack fine granularity for complex emotions. To address this, we propose \textbf{FreeAvatar}, a robust facial animation transfer method that relies solely on our learned expression representation. Specifically, FreeAvatar consists of two main components: the expression foundation model and the facial animation transfer model. In the first component, we initially construct a facial feature space through a face reconstruction task and then optimize the expression feature space by exploring the similarities among different expressions. Benefiting from training on the amounts of unlabeled facial images and re-collected expression comparison dataset, our model adapts freely and effectively to any in-the-wild input facial images. In the facial animation transfer component, we propose a novel Expression-driven Multi-avatar Animator, which first maps expressive semantics to the facial control parameters of 3D avatars and then imposes perceptual constraints between the input and output images to maintain expression consistency. To make the entire process differentiable, we employ a trained neural renderer to translate rig parameters into corresponding images. Furthermore, unlike previous methods that require separate decoders for each avatar, we propose a dynamic identity injection module that allows for the joint training of multiple avatars within a single network.
Abstract (translated)
视频驱动的3D面部动画传输旨在将虚拟角色复制演员的表情。现有的方法通过约束几何和感知一致性取得了显著的成果。然而，约束面部特征点的几何限制（如设计在面部特征点上的限制）不足以捕捉细微的情感，而经过分类任务训练的表情特征缺乏复杂情感的细粒度。为了解决这个问题，我们提出了FreeAvatar，一种仅依赖我们所学到的表情表示的稳健面部动画传输方法。具体来说，FreeAvatar包括两个主要组件：表情基础模型和面部动画传输模型。在第一个组件中，我们通过面部重建任务首先构建了一个面部特征空间，然后通过探索不同表情之间的相似性来优化表情特征空间。得益于在未标注面部图像的训练和重新收集表情比较数据集上训练，我们的模型可以自由地适应任何野外输入面部图像。在面部动画传输组件中，我们提出了一个新颖的表达驱动多虚拟角色动画器，它首先将表情语义映射到3D虚拟角色的面部控制参数上，然后对输入和输出图像之间的感知约束进行约束，以保持表情一致性。为了使整个过程具有可导性，我们使用训练好的神经渲染器将 rig 参数转换为相应的图像。此外，与之前的方法不同，我们提出了一个动态身份注入模块，允许在单个网络中联合训练多个虚拟角色。
URL
https://arxiv.org/abs/2409.13180
PDF
-
DreamHead: Learning Spatial-Temporal Correspondence via Hierarchical Diffusion for Audio-driven Talking Head Synthesis
2024-09-16 13:44:20Fa-Ting Hong, Yunfei Liu, Yu Li, Changyin Zhou, Fei Yu, Dan XuAbstract
Audio-driven talking head synthesis strives to generate lifelike video portraits from provided audio. The diffusion model, recognized for its superior quality and robust generalization, has been explored for this task. However, establishing a robust correspondence between temporal audio cues and corresponding spatial facial expressions with diffusion models remains a significant challenge in talking head generation. To bridge this gap, we present DreamHead, a hierarchical diffusion framework that learns spatial-temporal correspondences in talking head synthesis without compromising the model's intrinsic quality and adaptability.~DreamHead learns to predict dense facial landmarks from audios as intermediate signals to model the spatial and temporal correspondences.~Specifically, a first hierarchy of audio-to-landmark diffusion is first designed to predict temporally smooth and accurate landmark sequences given audio sequence signals. Then, a second hierarchy of landmark-to-image diffusion is further proposed to produce spatially consistent facial portrait videos, by modeling spatial correspondences between the dense facial landmark and appearance. Extensive experiments show that proposed DreamHead can effectively learn spatial-temporal consistency with the designed hierarchical diffusion and produce high-fidelity audio-driven talking head videos for multiple identities.
Abstract (translated)
音频驱动的讲话头合成旨在从提供的音频中生成逼真的视频肖像。扩散模型因其高质量和稳健的泛化能力而受到广泛探索。然而，在扩散模型中建立时空音频线索与相应面部表情的可靠对应仍然是一个重要的挑战。为了弥合这个差距，我们提出了DreamHead，一种分层扩散框架，它在不损害模型固有质量和适应性的前提下，从讲话头合成中学习时空对应关系。DreamHead从音频中预测丰富的面部关键点作为中间信号，以建模时空和面部表情的对应关系。具体来说，我们首先设计了一个音频到关键点的扩散层次结构，用于预测给定音频序列信号的时变平滑和准确的关键点序列。然后，我们进一步提出了一个地标到图像的扩散层次结构，通过建模密集面部关键点和外观之间的空间对应关系，产生具有空间一致性的面部肖像视频。大量实验证明，所提出的DreamHead可以有效地利用设计的分层扩散学习时空和面部表情的对应关系，并为多个身份生成高保真的音频驱动讲话头视频。
URL
https://arxiv.org/abs/2409.10281
PDF
-
KAN-Based Fusion of Dual-Domain for Audio-Driven Facial Landmarks Generation
2024-09-09 05:20:02Hoang-Son Vo-Thanh, Quang-Vinh Nguyen, Soo-Hyung KimAbstract
Audio-driven talking face generation is a widely researched topic due to its high applicability. Reconstructing a talking face using audio significantly contributes to fields such as education, healthcare, online conversations, virtual assistants, and virtual reality. Early studies often focused solely on changing the mouth movements, which resulted in outcomes with limited practical applications. Recently, researchers have proposed a new approach of constructing the entire face, including face pose, neck, and shoulders. To achieve this, they need to generate through landmarks. However, creating stable landmarks that align well with the audio is a challenge. In this paper, we propose the KFusion of Dual-Domain model, a robust model that generates landmarks from audio. We separate the audio into two distinct domains to learn emotional information and facial context, then use a fusion mechanism based on the KAN model. Our model demonstrates high efficiency compared to recent models. This will lay the groundwork for the development of the audio-driven talking face generation problem in the future.
Abstract (translated)
音频驱动的对话式面部生成是一个广泛研究的话题，因为它的实际应用非常广泛。通过音频重构对话式面部在教育、医疗、在线对话、虚拟助手和虚拟现实等领域具有重要意义。早期的研究通常仅关注改变嘴部的运动，导致实际应用效果有限。近年来，研究人员提出了一种新的方法来构建整个面部，包括脸部姿势、脖子和肩膀。要实现这一点，他们需要通过地标进行生成。然而，创建与音频一致的对齐稳定的地标是一个挑战。在本文中，我们提出了KFusion双域模型，一种基于音频生成地标的有前途的模型。我们将音频分为两个不同的领域来学习情感信息和面部上下文，然后使用基于KAN模型的融合机制。我们的模型在最近模型的效率方面表现出优异的表现。这将为实现音频驱动对话式面部生成问题的未来发展奠定了基础。
URL
https://arxiv.org/abs/2409.05330
PDF
-
T-FAKE: Synthesizing Thermal Images for Facial Landmarking
2024-08-27 15:07:58Philipp Flotho (Systems Neuroscience & Neurotechnology Unit, Faculty of Medicine, Saarland University & htw saar), Moritz Piening (Institute of Mathematics, Technische Universit\"at Berlin), Anna Kukleva (Max Planck Institute for Informatics, Saarland Informatics Campus), Gabriele Steidl (Institute of Mathematics, Technische Universit\"at Berlin)Abstract
Facial analysis is a key component in a wide range of applications such as security, autonomous driving, entertainment, and healthcare. Despite the availability of various facial RGB datasets, the thermal modality, which plays a crucial role in life sciences, medicine, and biometrics, has been largely overlooked. To address this gap, we introduce the T-FAKE dataset, a new large-scale synthetic thermal dataset with sparse and dense landmarks. To facilitate the creation of the dataset, we propose a novel RGB2Thermal loss function, which enables the transfer of thermal style to RGB faces. By utilizing the Wasserstein distance between thermal and RGB patches and the statistical analysis of clinical temperature distributions on faces, we ensure that the generated thermal images closely resemble real samples. Using RGB2Thermal style transfer based on our RGB2Thermal loss function, we create the T-FAKE dataset, a large-scale synthetic thermal dataset of faces. Leveraging our novel T-FAKE dataset, probabilistic landmark prediction, and label adaptation networks, we demonstrate significant improvements in landmark detection methods on thermal images across different landmark conventions. Our models show excellent performance with both sparse 70-point landmarks and dense 478-point landmark annotations. Our code and models are available at this https URL.
Abstract (translated)
面部分析是许多应用的关键组件，如安全、自动驾驶、娱乐和医疗。尽管各种面部RGB数据集已经存在，但生物医学和生物测量学中扮演关键角色的热模量（thermal modality）却受到了很大的忽视。为了填补这一空白，我们引入了T-FAKE数据集，一个大型合成热数据集，具有稀疏和密集的标志。为了方便数据集的创建，我们提出了一个新颖的RGB2Thermal损失函数，该函数可以将热风格传递到RGB面部。通过利用热和RGB补丁之间的Wasserstein距离和面部临床温度分布的统计分析，我们确保生成的热图像与真实样本非常相似。基于我们的RGB2Thermal损失函数，我们创建了T-FAKE数据集，一个大型合成热数据集。通过利用我们新颖的T-FAKE数据集、概率地标预测和标签适应网络，我们在不同地标范式下显著提高了热图像中的地标检测方法。我们的模型在稀疏70个地标和密集478个地标注释上表现优异。我们的代码和模型可在此处访问：https://www.thisurl.com/
URL
https://arxiv.org/abs/2408.15127
PDF
-
Ensemble Modeling of Multiple Physical Indicators to Dynamically Phenotype Autism Spectrum Disorder
2024-08-23 17:55:58Marie Huynh (Stanford University), Aaron Kline (Stanford University), Saimourya Surabhi (Stanford University), Kaitlyn Dunlap (Stanford University), Onur Cezmi Mutlu (Stanford University), Mohammadmahdi Honarmand (Stanford University), Parnian Azizian (Stanford University), Peter Washington (University of Hawaii at Manoa), Dennis P. Wall (Stanford University)Abstract
Early detection of autism, a neurodevelopmental disorder marked by social communication challenges, is crucial for timely intervention. Recent advancements have utilized naturalistic home videos captured via the mobile application GuessWhat. Through interactive games played between children and their guardians, GuessWhat has amassed over 3,000 structured videos from 382 children, both diagnosed with and without Autism Spectrum Disorder (ASD). This collection provides a robust dataset for training computer vision models to detect ASD-related phenotypic markers, including variations in emotional expression, eye contact, and head movements. We have developed a protocol to curate high-quality videos from this dataset, forming a comprehensive training set. Utilizing this set, we trained individual LSTM-based models using eye gaze, head positions, and facial landmarks as input features, achieving test AUCs of 86%, 67%, and 78%, respectively. To boost diagnostic accuracy, we applied late fusion techniques to create ensemble models, improving the overall AUC to 90%. This approach also yielded more equitable results across different genders and age groups. Our methodology offers a significant step forward in the early detection of ASD by potentially reducing the reliance on subjective assessments and making early identification more accessibly and equitable.
Abstract (translated)
早期对自闭症(ASD)的检测,这是一种以社交沟通困难为特征的神经发育障碍,对及时干预至关重要。最近的研究利用了通过移动应用程序GuessWhat捕捉的自然家庭视频,通过儿童和监护人之间的互动游戏累积了超过3,000个结构化视频,这些视频来自被确诊为ASD和未确诊ASD的382名儿童。这个数据集为训练计算机视觉模型检测ASD相关表型提供了有力的数据,包括情感表达、眼神接触和头部运动的变异性。我们开发了一个从该数据集中筛选高质量视频的协议,形成了一个全面的训练集。利用这个集,我们使用眼部注视、头部位置和面部关键点作为输入特征训练了基于LSTM的个体模型,实现了86%、67%和78%的测试AUC。为了提高诊断准确性,我们应用了晚融合技术创建了集成模型,提高了整体AUC至90%。这种方法还产生了更平等的结果,跨越了不同性别和年龄组。我们的方法在通过可能减少对主观评估的依赖来显著提高ASD早期诊断方面迈出了重要的一步,使早期识别更加容易和公平。
URL
https://arxiv.org/abs/2408.13255
PDF
-
G$^2$V$^2$former: Graph Guided Video Vision Transformer for Face Anti-Spoofing
2024-08-14 17:22:41Jingyi Yang, Zitong Yu, Xiuming Ni, Jia He, Hui LiAbstract
In videos containing spoofed faces, we may uncover the spoofing evidence based on either photometric or dynamic abnormality, even a combination of both. Prevailing face anti-spoofing (FAS) approaches generally concentrate on the single-frame scenario, however, purely photometric-driven methods overlook the dynamic spoofing clues that may be exposed over time. This may lead FAS systems to conclude incorrect judgments, especially in cases where it is easily distinguishable in terms of dynamics but challenging to discern in terms of photometrics. To this end, we propose the Graph Guided Video Vision Transformer (G$^2$V$^2$former), which combines faces with facial landmarks for photometric and dynamic feature fusion. We factorize the attention into space and time, and fuse them via a spatiotemporal block. Specifically, we design a novel temporal attention called Kronecker temporal attention, which has a wider receptive field, and is beneficial for capturing dynamic information. Moreover, we leverage the low-semantic motion of facial landmarks to guide the high-semantic change of facial expressions based on the motivation that regions containing landmarks may reveal more dynamic clues. Extensive experiments on nine benchmark datasets demonstrate that our method achieves superior performance under various scenarios. The codes will be released soon.
Abstract (translated)
在包含伪造脸的视频中，我们可能根据光电特征或动态异常来揭示伪造证据，甚至可以是两者之间的组合。预先存在的面部抗伪造（FAS）方法通常集中于单帧场景，然而，仅基于光学的伪造证据可能掩盖了随着时间的推移可能暴露的动态伪造线索。这可能导致FAS系统得出错误的判断，尤其是在 dynamics 上容易区分，但在 photometrics 上很难分辨的情况下。因此，我们提出了Graph Guided Video Vision Transformer (G$^2$V$^2$Transformer)，它结合了面部特征和面部标志以进行光电和动态特征融合。我们将注意力因素化到空间和时间，并通过一个spatiotemporal块将它们融合。具体来说，我们设计了一个名为Kronecker temporal attention的新型时间注意力，它具有更宽的接收范围，对于捕捉动态信息非常有利。此外，我们利用面部标志的低语义运动来指导根据动机，含有标志的区域可能揭示更动态的提示的脸上表情的语义变化。在九个基准数据集上的大量实验证明，我们的方法在各种情景下取得了卓越的性能。代码即将发布。
URL
https://arxiv.org/abs/2408.07675
PDF
-
Real-Time Drowsiness Detection Using Eye Aspect Ratio and Facial Landmark Detection
2024-08-11 17:34:24Varun Shiva Krishna Rupani, Velpooru Venkata Sai Thushar, Kondadi TejithAbstract
Drowsiness detection is essential for improving safety in areas such as transportation and workplace health. This study presents a real-time system designed to detect drowsiness using the Eye Aspect Ratio (EAR) and facial landmark detection techniques. The system leverages Dlibs pre-trained shape predictor model to accurately detect and monitor 68 facial landmarks, which are used to compute the EAR. By establishing a threshold for the EAR, the system identifies when eyes are closed, indicating potential drowsiness. The process involves capturing a live video stream, detecting faces in each frame, extracting eye landmarks, and calculating the EAR to assess alertness. Our experiments show that the system reliably detects drowsiness with high accuracy while maintaining low computational demands. This study offers a strong solution for real-time drowsiness detection, with promising applications in driver monitoring and workplace safety. Future research will investigate incorporating additional physiological and contextual data to further enhance detection accuracy and reliability.
Abstract (translated)
昏睡检测对于改善交通和工作场所的健康安全至关重要。这项研究介绍了一种实时系统，用于通过Eye Aspect Ratio（EAR）和面部 landmark 检测技术检测昏睡。该系统利用了Dlibs预训练的形状预测模型，准确检测并监控68个面部表情标记物，这些标记物用于计算EAR。通过建立EAR的阈值，系统在眼睛关闭时识别眼睛，表明存在昏睡。该过程涉及捕获实时视频流，检测每帧的面部，提取眼部标记物，并计算EAR以评估警觉性。我们的实验结果表明，该系统在保持低计算负担的同时，以高准确度可靠地检测昏睡。这项研究为实时昏睡检测提供了一个强大的解决方案，在驾驶员监控和职场安全方面具有光明的前景。未来研究将调查引入其他生理学和上下文数据，以进一步增强检测准确性和可靠性。
URL
https://arxiv.org/abs/2408.05836
PDF
