Facial_Landmark
-
Phoneme-Level Visual Speech Recognition via Point-Visual Fusion and Language Model Reconstruction
2025-07-25 00:38:39Matthew Kit Khinn Teng, Haibo Zhang, Takeshi SaitohAbstract
Visual Automatic Speech Recognition (V-ASR) is a challenging task that involves interpreting spoken language solely from visual information, such as lip movements and facial expressions. This task is notably challenging due to the absence of auditory cues and the visual ambiguity of phonemes that exhibit similar visemes-distinct sounds that appear identical in lip motions. Existing methods often aim to predict words or characters directly from visual cues, but they commonly suffer from high error rates due to viseme ambiguity and require large amounts of pre-training data. We propose a novel phoneme-based two-stage framework that fuses visual and landmark motion features, followed by an LLM model for word reconstruction to address these challenges. Stage 1 consists of V-ASR, which outputs the predicted phonemes, thereby reducing training complexity. Meanwhile, the facial landmark features address speaker-specific facial characteristics. Stage 2 comprises an encoder-decoder LLM model, NLLB, that reconstructs the output phonemes back to words. Besides using a large visual dataset for deep learning fine-tuning, our PV-ASR method demonstrates superior performance by achieving 17.4% WER on the LRS2 and 21.0% WER on the LRS3 dataset.
Abstract (translated)
视觉自动语音识别（V-ASR）是一项具有挑战性的任务，它需要仅通过视觉信息（如唇部动作和面部表情）来解读口语。由于缺少听觉线索以及表现出类似视觉音素的发音在唇部运动中看起来相同造成的视觉模糊性，这项任务尤为困难。现有的方法通常旨在直接从视觉提示预测单词或字符，但它们往往因视觉音素的歧义而产生较高的错误率，并且需要大量的预训练数据。 我们提出了一种基于语音单元（音位）的两阶段框架，该框架融合了视觉和面部关键点运动特征，并随后使用LLM模型进行词汇重构以应对这些挑战。第一阶段包括V-ASR，输出预测的音素，从而减少训练复杂度。与此同时，面部关键点特征处理特定说话人的面部特性。第二阶段包含一个编码器-解码器LLM模型NLLB，用于将输出的音素重新构建成单词。 除了使用大规模视觉数据集进行深度学习微调外，我们的PV-ASR方法通过在LRS2和LRS3数据集中分别实现了17.4%和21.0%的词错误率（WER），展示了优于现有技术的表现。
URL
https://arxiv.org/abs/2507.18863
PDF
-
Deepfake Detection Via Facial Feature Extraction and Modeling
2025-07-24 21:30:51Benjamin Carter, Nathan Dilla, Micheal Callahan, Atuhaire AmbalaAbstract
The rise of deepfake technology brings forth new questions about the authenticity of various forms of media found online today. Videos and images generated by artificial intelligence (AI) have become increasingly more difficult to differentiate from genuine media, resulting in the need for new models to detect artificially-generated media. While many models have attempted to solve this, most focus on direct image processing, adapting a convolutional neural network (CNN) or a recurrent neural network (RNN) that directly interacts with the video image data. This paper introduces an approach of using solely facial landmarks for deepfake detection. Using a dataset consisting of both deepfake and genuine videos of human faces, this paper describes an approach for extracting facial landmarks for deepfake detection, focusing on identifying subtle inconsistencies in facial movements instead of raw image processing. Experimental results demonstrated that this feature extraction technique is effective in various neural network models, with the same facial landmarks tested on three neural network models, with promising performance metrics indicating its potential for real-world applications. The findings discussed in this paper include RNN and artificial neural network (ANN) models with accuracy between 96% and 93%, respectively, with a CNN model hovering around 78%. This research challenges the assumption that raw image processing is necessary to identify deepfake videos by presenting a facial feature extraction approach compatible with various neural network models while requiring fewer parameters.
Abstract (translated)
深度伪造技术的兴起引发了关于当今在线媒体真实性的新问题。由人工智能（AI）生成的视频和图像变得越来越难以与真实媒体区分开来，这促使了新模型的发展以检测人工生成的媒体。虽然已有许多模型试图解决这一问题，但大多数集中于直接处理图像，采用适应卷积神经网络（CNN）或递归神经网络（RNN），这些模型会直接操作视频图像数据。本文介绍了一种仅使用面部标志进行深度伪造检测的方法。通过使用包含真人脸深伪视频和真实视频的数据库，本文描述了提取用于深度伪造检测的面部标志的方法，并专注于识别面部运动中的细微不一致之处，而不是处理原始图像。 实验结果表明，这种特征提取技术在各种神经网络模型中都有效，同一批面部标志经过三种不同神经网络模型测试后，显示出令人鼓舞的表现指标，证明其具有潜在的实际应用价值。本文讨论的发现包括递归神经网络（RNN）和人工神经网络（ANN）模型分别达到了96%至93%的准确率，而卷积神经网络（CNN）模型则维持在78%左右。 这项研究通过提供一种与各种神经网络模型兼容的面部特征提取方法来挑战单纯依靠原始图像处理识别深度伪造视频这一假设，并且这种新方法所需的参数更少。
URL
https://arxiv.org/abs/2507.18815
PDF
-
NLML-HPE: Head Pose Estimation with Limited Data via Manifold Learning
2025-07-24 14:08:33Mahdi Ghafourian, Federico M. SuknoAbstract
Head pose estimation (HPE) plays a critical role in various computer vision applications such as human-computer interaction and facial recognition. In this paper, we propose a novel deep learning approach for head pose estimation with limited training data via non-linear manifold learning called NLML-HPE. This method is based on the combination of tensor decomposition (i.e., Tucker decomposition) and feed forward neural networks. Unlike traditional classification-based approaches, our method formulates head pose estimation as a regression problem, mapping input landmarks into a continuous representation of pose angles. To this end, our method uses tensor decomposition to split each Euler angle (yaw, pitch, roll) to separate subspaces and models each dimension of the underlying manifold as a cosine curve. We address two key challenges: 1. Almost all HPE datasets suffer from incorrect and inaccurate pose annotations. Hence, we generated a precise and consistent 2D head pose dataset for our training set by rotating 3D head models for a fixed set of poses and rendering the corresponding 2D images. 2. We achieved real-time performance with limited training data as our method accurately captures the nature of rotation of an object from facial landmarks. Once the underlying manifold for rotation around each axis is learned, the model is very fast in predicting unseen data. Our training and testing code is available online along with our trained models: https: //github.com/MahdiGhafoorian/NLML_HPE.
Abstract (translated)
头部姿态估计（HPE）在人机交互和面部识别等众多计算机视觉应用中扮演着关键角色。本文提出了一种新颖的深度学习方法，通过非线性流形学习（NLML-HPE），利用有限训练数据进行头部姿态估计。该方法基于张量分解（即Tucker分解）与前馈神经网络相结合。 不同于传统的分类方法，我们的方法将头部姿态估计视为回归问题，并将输入标记映射到连续的姿态角度表示中。为此，我们使用张量分解将每个欧拉角（偏航角、俯仰角和滚转角）拆分为独立的子空间，并将潜在流形的每一维度建模为余弦曲线。 本研究解决了两个主要挑战： 1. 几乎所有头部姿态数据集都存在错误或不准确的姿态标注。为此，我们通过旋转固定姿势集合中的3D头模型并渲染相应的2D图像来生成精确且一致的二维头部姿态训练数据集。 2. 我们在有限的训练数据情况下实现了实时性能，因为我们的方法能够从面部标记中精确捕捉物体旋转的本质特性。一旦学习了每个轴上的旋转潜在流形，该模型就能快速预测未见过的数据。 我们提供的训练和测试代码以及预训练模型可以在线获取：https://github.com/MahdiGhafoorian/NLML_HPE
URL
https://arxiv.org/abs/2507.18429
PDF
-
Evaluation of facial landmark localization performance in a surgical setting
2025-07-24 09:40:47Ines Frajtag, Marko \v{S}vaco, Filip \v{S}uligojAbstract
The use of robotics, computer vision, and their applications is becoming increasingly widespread in various fields, including medicine. Many face detection algorithms have found applications in neurosurgery, ophthalmology, and plastic surgery. A common challenge in using these algorithms is variable lighting conditions and the flexibility of detection positions to identify and precisely localize patients. The proposed experiment tests the MediaPipe algorithm for detecting facial landmarks in a controlled setting, using a robotic arm that automatically adjusts positions while the surgical light and the phantom remain in a fixed position. The results of this study demonstrate that the improved accuracy of facial landmark detection under surgical lighting significantly enhances the detection performance at larger yaw and pitch angles. The increase in standard deviation/dispersion occurs due to imprecise detection of selected facial landmarks. This analysis allows for a discussion on the potential integration of the MediaPipe algorithm into medical procedures.
Abstract (translated)
机器人技术、计算机视觉及其应用在包括医学在内的各个领域中越来越广泛。许多面部检测算法已经在神经外科、眼科和整形外科等领域找到了应用。使用这些算法的一个常见挑战是变化的照明条件以及为准确识别和定位患者而灵活调整检测位置的需求。本次实验在一个受控环境中测试了MediaPipe算法，用于在手术灯保持固定且模型头部静止的情况下，由机械臂自动调节不同角度来检测面部特征点。 该研究结果表明，在手术照明条件下改进后的面部特征点检测精度显著提高了偏航角（左右旋转）和俯仰角（上下倾斜）较大时的检测性能。标准偏差/离散度的增加是由于对选定面部特征点识别的不精确性所致。这项分析为MediaPipe算法在医疗程序中的潜在整合提供了讨论基础。
URL
https://arxiv.org/abs/2507.18248
PDF
-
Morpheus: A Neural-driven Animatronic Face with Hybrid Actuation and Diverse Emotion Control
2025-07-22 14:42:49Zongzheng Zhang, Jiawen Yang, Ziqiao Peng, Meng Yang, Jianzhu Ma, Lin Cheng, Huazhe Xu, Hang Zhao, Hao ZhaoAbstract
Previous animatronic faces struggle to express emotions effectively due to hardware and software limitations. On the hardware side, earlier approaches either use rigid-driven mechanisms, which provide precise control but are difficult to design within constrained spaces, or tendon-driven mechanisms, which are more space-efficient but challenging to control. In contrast, we propose a hybrid actuation approach that combines the best of both worlds. The eyes and mouth-key areas for emotional expression-are controlled using rigid mechanisms for precise movement, while the nose and cheek, which convey subtle facial microexpressions, are driven by strings. This design allows us to build a compact yet versatile hardware platform capable of expressing a wide range of emotions. On the algorithmic side, our method introduces a self-modeling network that maps motor actions to facial landmarks, allowing us to automatically establish the relationship between blendshape coefficients for different facial expressions and the corresponding motor control signals through gradient backpropagation. We then train a neural network to map speech input to corresponding blendshape controls. With our method, we can generate distinct emotional expressions such as happiness, fear, disgust, and anger, from any given sentence, each with nuanced, emotion-specific control signals-a feature that has not been demonstrated in earlier systems. We release the hardware design and code at this https URL and this https URL.
Abstract (translated)
之前的人形机器人面部在表达情感时由于硬件和软件的限制而显得不够有效。从硬件角度来看，早期的方法要么使用刚性驱动机制，这种方法能够提供精确控制但难以在有限的空间内设计；要么使用肌腱驱动机制，这种方式虽然更节省空间但控制起来更加困难。相比之下，我们提出了一种混合致动方法，结合了两种方式的优点。眼睛和嘴巴是情感表达的关键部位，这些区域采用刚性机制进行精细运动控制，而鼻子和脸颊则通过线缆驱动，以传达微妙的面部微表情。这种设计使我们可以构建一个既紧凑又多功能的硬件平台，能够表达广泛的情感。 在算法方面，我们的方法引入了一个自建模网络，该网络将电机动作映射到面部特征点上，并允许我们自动建立不同面部表情的变形形状系数与相应的电机控制信号之间的关系通过反向传播梯度实现。接着训练一个神经网络以从语音输入中映射出对应的变形形状控制器。借助我们的方法，可以从任何给定句子生成如快乐、恐惧、厌恶和愤怒等独特的表情，每个情感都具有细微且特定的情绪化控制信号——这是之前系统未曾展示过的特性。 我们将硬件设计和代码发布在这个链接：[此URL] 和这个链接：[此URL]（请将“https URL”替换为实际的链接地址）。
URL
https://arxiv.org/abs/2507.16645
PDF
-
Navigating Large-Pose Challenge for High-Fidelity Face Reenactment with Video Diffusion Model
2025-07-22 08:23:43Mingtao Guo, Guanyu Xing, Yanci Zhang, Yanli LiuAbstract
Face reenactment aims to generate realistic talking head videos by transferring motion from a driving video to a static source image while preserving the source identity. Although existing methods based on either implicit or explicit keypoints have shown promise, they struggle with large pose variations due to warping artifacts or the limitations of coarse facial landmarks. In this paper, we present the Face Reenactment Video Diffusion model (FRVD), a novel framework for high-fidelity face reenactment under large pose changes. Our method first employs a motion extractor to extract implicit facial keypoints from the source and driving images to represent fine-grained motion and to perform motion alignment through a warping module. To address the degradation introduced by warping, we introduce a Warping Feature Mapper (WFM) that maps the warped source image into the motion-aware latent space of a pretrained image-to-video (I2V) model. This latent space encodes rich priors of facial dynamics learned from large-scale video data, enabling effective warping correction and enhancing temporal coherence. Extensive experiments show that FRVD achieves superior performance over existing methods in terms of pose accuracy, identity preservation, and visual quality, especially in challenging scenarios with extreme pose variations.
Abstract (translated)
面部重演的目标是通过将驱动视频中的运动转移到静态源图像上，生成逼真的说话头像视频，同时保持源身份的完整性。尽管基于隐式或显式关键点的方法已经显示出潜力，但它们在处理大姿态变化时由于扭曲伪影或粗略面部标志的局限性而面临挑战。 在这篇论文中，我们介绍了Face Reenactment Video Diffusion模型（FRVD），这是一种用于处理大幅度姿势变化下的高质量面部重演的新颖框架。我们的方法首先采用了一个运动提取器来从源图像和驱动图像中隐式地提取面部关键点，以表示精细的运动并通过一个扭曲模块进行运动对齐。为了解决由于扭曲引入的质量下降问题，我们提出了一种Warped Feature Mapper（WFM），它将扭曲后的源图像映射到预训练的图像到视频(I2V)模型的感知运动潜在空间中。该潜在空间包含了从大规模视频数据中学到的人脸动态先验知识，这使得有效的扭曲纠正成为可能，并增强了时间一致性。 广泛的实验表明，在姿态准确性、身份保持和视觉质量方面，FRVD在包括极端姿势变化在内的具有挑战性的场景下都超越了现有的方法。
URL
https://arxiv.org/abs/2507.16341
PDF
-
A Lightweight Face Quality Assessment Framework to Improve Face Verification Performance in Real-Time Screening Applications
2025-07-21 18:04:14Ahmed Aman Ibrahim, Hamad Mansour Alawar, Abdulnasser Abbas Zehi, Ahmed Mohammad Alkendi, Bilal Shafi Ashfaq Ahmed Mirza, Shan Ullah, Ismail Lujain Jaleel, Hassan UgailAbstract
Face image quality plays a critical role in determining the accuracy and reliability of face verification systems, particularly in real-time screening applications such as surveillance, identity verification, and access control. Low-quality face images, often caused by factors such as motion blur, poor lighting conditions, occlusions, and extreme pose variations, significantly degrade the performance of face recognition models, leading to higher false rejection and false acceptance rates. In this work, we propose a lightweight yet effective framework for automatic face quality assessment, which aims to pre-filter low-quality face images before they are passed to the verification pipeline. Our approach utilises normalised facial landmarks in conjunction with a Random Forest Regression classifier to assess image quality, achieving an accuracy of 96.67\%. By integrating this quality assessment module into the face verification process, we observe a substantial improvement in performance, including a comfortable 99.7\% reduction in the false rejection rate and enhanced cosine similarity scores when paired with the ArcFace face verification model. To validate our approach, we have conducted experiments on a real-world dataset collected comprising over 600 subjects captured from CCTV footage in unconstrained environments within Dubai Police. Our results demonstrate that the proposed framework effectively mitigates the impact of poor-quality face images, outperforming existing face quality assessment techniques while maintaining computational efficiency. Moreover, the framework specifically addresses two critical challenges in real-time screening: variations in face resolution and pose deviations, both of which are prevalent in practical surveillance scenarios.
Abstract (translated)
面部图像质量在决定人脸识别系统的准确性和可靠性方面起着关键作用，尤其是在监控、身份验证和访问控制等实时筛选应用中。低质量的面部图像通常由运动模糊、光照条件差、遮挡以及极端姿态变化等因素引起，这会显著降低人脸识别模型的表现，导致更高的误拒率和误识率。在本研究中，我们提出了一种轻量级但有效的自动面部质量评估框架，旨在筛选出低质量的面部图像，然后再将其传递给身份验证流程。我们的方法利用标准化的面部特征点与随机森林回归分类器结合来评估图像质量，达到了96.67%的准确率。通过将这种质量评估模块集成到人脸识别过程中，我们观察到了显著的性能提升，包括误拒率下降了99.7%，并与ArcFace人脸识别模型搭配使用时，余弦相似度评分也有所提高。 为了验证我们的方法，我们在一个包含超过600名受试者的真实世界数据集上进行了实验，该数据集由迪拜警察在不受控环境中从闭路电视录像中采集。我们实验证明，提出的框架有效减轻了低质量面部图像的影响，在保持计算效率的同时超越了现有的面部质量评估技术。此外，本框架特别解决了实时筛选中的两个关键挑战：面部分辨率的变化和姿态的偏移，这两种情况在实际监控场景中普遍存在。
URL
https://arxiv.org/abs/2507.15961
PDF
-
Rethinking Occlusion in FER: A Semantic-Aware Perspective and Go Beyond
2025-07-21 09:04:29Huiyu Zhai, Xingxing Yang, Yalan Ye, Chenyang Li, Bin Fan, Changze LiAbstract
Facial expression recognition (FER) is a challenging task due to pervasive occlusion and dataset biases. Especially when facial information is partially occluded, existing FER models struggle to extract effective facial features, leading to inaccurate classifications. In response, we present ORSANet, which introduces the following three key contributions: First, we introduce auxiliary multi-modal semantic guidance to disambiguate facial occlusion and learn high-level semantic knowledge, which is two-fold: 1) we introduce semantic segmentation maps as dense semantics prior to generate semantics-enhanced facial representations; 2) we introduce facial landmarks as sparse geometric prior to mitigate intrinsic noises in FER, such as identity and gender biases. Second, to facilitate the effective incorporation of these two multi-modal priors, we customize a Multi-scale Cross-interaction Module (MCM) to adaptively fuse the landmark feature and semantics-enhanced representations within different scales. Third, we design a Dynamic Adversarial Repulsion Enhancement Loss (DARELoss) that dynamically adjusts the margins of ambiguous classes, further enhancing the model's ability to distinguish similar expressions. We further construct the first occlusion-oriented FER dataset to facilitate specialized robustness analysis on various real-world occlusion conditions, dubbed Occlu-FER. Extensive experiments on both public benchmarks and Occlu-FER demonstrate that our proposed ORSANet achieves SOTA recognition performance. Code is publicly available at this https URL.
Abstract (translated)
面部表情识别（FER）是一个具有挑战性的任务，由于普遍存在遮挡和数据集偏差的问题。特别是在脸部信息部分被遮挡的情况下，现有的面部表情识别模型难以提取有效的面部特征，导致分类不准确。为此，我们提出了ORSANet，并引入了以下三个关键贡献： 第一，我们引入了辅助多模态语义引导来区分面部遮挡并学习高层次的语义知识，具体来说： 1）我们将语义分割图作为稠密语义先验以生成增强语义的面部表示； 2）我们将面部特征点（landmarks）作为稀疏几何先验来缓解FER中的固有噪声问题，如身份和性别偏差。 第二，为了有效地整合这两种多模态先验，我们定制了一种多尺度交叉交互模块（MCM），该模块可以在不同尺度下自适应地融合特征点特征与增强语义的表示。 第三，我们设计了一种动态对抗排斥强化损失函数（DARELoss），该函数可以动态调整模糊类别之间的边界，进一步增强了模型区分相似表情的能力。此外，为了进行特定的鲁棒性分析，我们在各种真实世界遮挡条件下构建了首个面向遮挡的面部表情识别数据集，命名为Occlu-FER。 在公开基准测试和Occlu-FER上的广泛实验表明，我们提出的ORSANet达到了最先进的（SOTA）识别性能。代码可在提供的链接中找到。
URL
https://arxiv.org/abs/2507.15401
PDF
-
Exp-Graph: How Connections Learn Facial Attributes in Graph-based Expression Recognition
2025-07-19 13:10:21Nandani Sharma, Dinesh SinghAbstract
Facial expression recognition is crucial for human-computer interaction applications such as face animation, video surveillance, affective computing, medical analysis, etc. Since the structure of facial attributes varies with facial expressions, incorporating structural information into facial attributes is essential for facial expression recognition. In this paper, we propose Exp-Graph, a novel framework designed to represent the structural relationships among facial attributes using graph-based modeling for facial expression recognition. For facial attributes graph representation, facial landmarks are used as the graph's vertices. At the same time, the edges are determined based on the proximity of the facial landmark and the similarity of the local appearance of the facial attributes encoded using the vision transformer. Additionally, graph convolutional networks are utilized to capture and integrate these structural dependencies into the encoding of facial attributes, thereby enhancing the accuracy of expression recognition. Thus, Exp-Graph learns from the facial attribute graphs highly expressive semantic representations. On the other hand, the vision transformer and graph convolutional blocks help the framework exploit the local and global dependencies among the facial attributes that are essential for the recognition of facial expressions. We conducted comprehensive evaluations of the proposed Exp-Graph model on three benchmark datasets: Oulu-CASIA, eNTERFACE05, and AFEW. The model achieved recognition accuracies of 98.09\%, 79.01\%, and 56.39\%, respectively. These results indicate that Exp-Graph maintains strong generalization capabilities across both controlled laboratory settings and real-world, unconstrained environments, underscoring its effectiveness for practical facial expression recognition applications.
Abstract (translated)
面部表情识别对于人脸动画、视频监控、情感计算、医学分析等人机交互应用至关重要。由于面部特征结构随面部表情变化而改变，因此将结构信息融入到面部属性中对表情识别而言非常关键。在本文中，我们提出了Exp-Graph，这是一种新颖的框架，旨在利用基于图模型的方法来表示面部属性之间的结构关系，以进行面部表情识别。对于面部属性图表示，我们将面部特征点用作图的顶点；同时，边根据面部特征点的接近度以及使用视觉变换器编码的局部外观相似性确定。此外，还应用了图卷积网络（Graph Convolutional Networks, GCNs）来捕捉并整合这些结构依赖关系到面部属性的编码中，从而提高表情识别的准确性。因此，Exp-Graph从面部属性图中学到了高度表达性的语义表示。另一方面，视觉变换器和图卷积块帮助框架利用了面部属性之间的局部和全局依赖性，这对于面部表情识别至关重要。 我们在三个基准数据集上对提出的Exp-Graph模型进行了全面评估：Oulu-CASIA、eNTERFACE05 和 AFEW。该模型分别取得了98.09%、79.01% 和 56.39% 的识别准确率。这些结果表明，Exp-Graph 在受控实验室环境和现实世界的无约束环境中均保持了强大的泛化能力，这强调了其在实际面部表情识别应用中的有效性。
URL
https://arxiv.org/abs/2507.14608
PDF
-
FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformers
2025-07-17 09:50:43Qiang Wang, Mengchao Wang, Fan Jiang, Yaqi Fan, Yonggang Qi, Mu XuAbstract
Producing expressive facial animations from static images is a challenging task. Prior methods relying on explicit geometric priors (e.g., facial landmarks or 3DMM) often suffer from artifacts in cross reenactment and struggle to capture subtle emotions. Furthermore, existing approaches lack support for multi-character animation, as driving features from different individuals frequently interfere with one another, complicating the task. To address these challenges, we propose FantasyPortrait, a diffusion transformer based framework capable of generating high-fidelity and emotion-rich animations for both single- and multi-character scenarios. Our method introduces an expression-augmented learning strategy that utilizes implicit representations to capture identity-agnostic facial dynamics, enhancing the model's ability to render fine-grained emotions. For multi-character control, we design a masked cross-attention mechanism that ensures independent yet coordinated expression generation, effectively preventing feature interference. To advance research in this area, we propose the Multi-Expr dataset and ExprBench, which are specifically designed datasets and benchmarks for training and evaluating multi-character portrait animations. Extensive experiments demonstrate that FantasyPortrait significantly outperforms state-of-the-art methods in both quantitative metrics and qualitative evaluations, excelling particularly in challenging cross reenactment and multi-character contexts. Our project page is this https URL.
Abstract (translated)
从静态图像生成富有表现力的面部动画是一项具有挑战性的任务。以往依赖显式几何先验（如面部特征点或3DMM）的方法在跨场景重现时常常产生伪影，并且难以捕捉细微的情感变化。此外，现有的方法缺乏对多角色动画的支持，因为不同个体驱动特征之间的相互干扰会使得任务复杂化。为了应对这些挑战，我们提出了FantasyPortrait，这是一种基于扩散变换器的框架，能够为单角色和多角色场景生成高保真度、情感丰富的动画。我们的方法引入了一种增强表情的学习策略，利用隐式表示来捕捉无关身份的脸部动态变化，从而增强了模型渲染细微情绪的能力。对于多角色控制，我们设计了一种掩码交叉注意力机制，确保各个表达的独立性和协调性，有效防止特征干扰。为了推进该领域的研究，我们提出了Multi-Expr数据集和ExprBench，这是专门为训练和评估多角色肖像动画而特别设计的数据集和基准测试。广泛的实验表明，FantasyPortrait在定量指标和定性评价中均显著优于当前最先进的方法，在具有挑战性的跨场景重现和多角色情景下尤其表现出色。 我们的项目页面在此链接：https://这个URL提供具体信息和资源。
URL
https://arxiv.org/abs/2507.12956
PDF
-
ATL-Diff: Audio-Driven Talking Head Generation with Early Landmarks-Guide Noise Diffusion
2025-07-17 05:40:51Hoang-Son Vo, Quang-Vinh Nguyen, Seungwon Kim, Hyung-Jeong Yang, Soonja Yeom, Soo-Hyung KimAbstract
Audio-driven talking head generation requires precise synchronization between facial animations and audio signals. This paper introduces ATL-Diff, a novel approach addressing synchronization limitations while reducing noise and computational costs. Our framework features three key components: a Landmark Generation Module converting audio to facial landmarks, a Landmarks-Guide Noise approach that decouples audio by distributing noise according to landmarks, and a 3D Identity Diffusion network preserving identity characteristics. Experiments on MEAD and CREMA-D datasets demonstrate that ATL-Diff outperforms state-of-the-art methods across all metrics. Our approach achieves near real-time processing with high-quality animations, computational efficiency, and exceptional preservation of facial nuances. This advancement offers promising applications for virtual assistants, education, medical communication, and digital platforms. The source code is available at: \href{this https URL}{this https URL}
Abstract (translated)
基于音频的面部动画生成需要面部动画与音频信号之间精确同步。本文介绍了一种名为ATL-Diff的新方法，该方法解决了同步限制问题，并在减少噪声和计算成本方面表现出色。我们的框架包含三个关键组件：一个将音频转换为面部标志的关键点生成模块、一个通过根据面部标志分布噪声来解耦音频的Landmarks-Guide噪声方法以及一个保留身份特征的3D Identity Diffusion网络。 我们在MEAD和CREMA-D数据集上的实验表明，ATL-Diff在所有评估指标上均优于现有最先进的方法。我们的方法实现了接近实时处理、高质量动画生成、计算效率，并且在面部细节保留方面表现出色。这项进步为虚拟助手、教育、医疗沟通和数字平台的应用提供了广阔前景。 源代码可以在以下网址获取：[此处提供具体的URL链接]
URL
https://arxiv.org/abs/2507.12804
PDF
-
De-Fake: Style based Anomaly Deepfake Detection
2025-07-04 06:42:51Sudev Kumar Padhi, Harshit Kumar, Umesh Kashyap, Sk. Subidh AliAbstract
Detecting deepfakes involving face-swaps presents a significant challenge, particularly in real-world scenarios where anyone can perform face-swapping with freely available tools and apps without any technical knowledge. Existing deepfake detection methods rely on facial landmarks or inconsistencies in pixel-level features and often struggle with face-swap deepfakes, where the source face is seamlessly blended into the target image or video. The prevalence of face-swap is evident in everyday life, where it is used to spread false information, damage reputations, manipulate political opinions, create non-consensual intimate deepfakes (NCID), and exploit children by enabling the creation of child sexual abuse material (CSAM). Even prominent public figures are not immune to its impact, with numerous deepfakes of them circulating widely across social media platforms. Another challenge faced by deepfake detection methods is the creation of datasets that encompass a wide range of variations, as training models require substantial amounts of data. This raises privacy concerns, particularly regarding the processing and storage of personal facial data, which could lead to unauthorized access or misuse. Our key idea is to identify these style discrepancies to detect face-swapped images effectively without accessing the real facial image. We perform comprehensive evaluations using multiple datasets and face-swapping methods, which showcases the effectiveness of SafeVision in detecting face-swap deepfakes across diverse scenarios. SafeVision offers a reliable and scalable solution for detecting face-swaps in a privacy preserving manner, making it particularly effective in challenging real-world applications. To the best of our knowledge, SafeVision is the first deepfake detection using style features while providing inherent privacy protection.
Abstract (translated)
检测涉及面部交换的深度伪造（Deepfakes）在现实场景中面临巨大挑战，因为任何人都可以使用免费工具和应用程序进行面部交换，而无需任何技术知识。现有的深度伪造检测方法依赖于面部特征点或像素级不一致来识别，但对于无缝融合源面孔到目标图像或视频中的面部交换深度伪造，这些方法往往效果不佳。在日常生活中，这种类型的面部交换被用于传播虚假信息、损害声誉、操纵政治观点、创建非自愿亲密深度伪造（NCID）以及通过允许生成儿童性虐待材料（CSAM）来剥削儿童。即使是知名公众人物也无法幸免于其影响，在社交媒体平台上广泛流传着他们的许多深度伪造作品。另一个挑战是为涵盖各种变化的训练模型创建数据集，这引发了有关处理和存储个人面部数据的隐私担忧，可能会导致未经授权访问或滥用。 我们的主要想法是在不获取真实面部图像的情况下，通过识别这些风格差异来有效检测面部交换图片。我们使用多种数据集和面部交换方法进行了全面评估，展示了SafeVision在各种场景下检测面部交换深度伪造的有效性。SafeVision提供了一种可靠且可扩展的解决方案，在保护隐私的同时检测面部交换，使其特别适用于具有挑战性的现实应用。 据我们所知，SafeVision是第一个利用风格特征进行深度伪造检测的方法，并提供了固有的隐私保护功能。
URL
https://arxiv.org/abs/2507.03334
PDF
-
Facial Landmark Visualization and Emotion Recognition Through Neural Networks
2025-06-20 17:45:34Israel Ju\'arez-Jim\'enez, Tiffany Guadalupe Mart\'inez Paredes, Jes\'us Garc\'ia-Ram\'irez, Eric Ramos AguilarAbstract
Emotion recognition from facial images is a crucial task in human-computer interaction, enabling machines to learn human emotions through facial expressions. Previous studies have shown that facial images can be used to train deep learning models; however, most of these studies do not include a through dataset analysis. Visualizing facial landmarks can be challenging when extracting meaningful dataset insights; to address this issue, we propose facial landmark box plots, a visualization technique designed to identify outliers in facial datasets. Additionally, we compare two sets of facial landmark features: (i) the landmarks' absolute positions and (ii) their displacements from a neutral expression to the peak of an emotional expression. Our results indicate that a neural network achieves better performance than a random forest classifier.
Abstract (translated)
面部图像的情感识别是人机交互中的一个关键任务，它使机器能够通过面部表情来学习人类的情绪。以往的研究表明，可以使用面部图像来训练深度学习模型；然而，大多数这些研究并未进行彻底的数据集分析。在提取有意义的数据集洞察时，可视化面部特征点可能会具有挑战性；为了解决这个问题，我们提出了面部特征框图技术，这是一种用于识别面部数据集中异常值的可视化方法。此外，我们将两组面部特征点进行了比较：（i）特征点的绝对位置和（ii）从中性表情到情感顶峰的表情变化中的位移。我们的研究结果表明，神经网络的表现优于随机森林分类器。
URL
https://arxiv.org/abs/2506.17191
PDF
-
MOL: Joint Estimation of Micro-Expression, Optical Flow, and Landmark via Transformer-Graph-Style Convolution
2025-06-17 13:35:06Zhiwen Shao, Yifan Cheng, Feiran Li, Yong Zhou, Xuequan Lu, Yuan Xie, Lizhuang MaAbstract
Facial micro-expression recognition (MER) is a challenging problem, due to transient and subtle micro-expression (ME) actions. Most existing methods depend on hand-crafted features, key frames like onset, apex, and offset frames, or deep networks limited by small-scale and low-diversity datasets. In this paper, we propose an end-to-end micro-action-aware deep learning framework with advantages from transformer, graph convolution, and vanilla convolution. In particular, we propose a novel F5C block composed of fully-connected convolution and channel correspondence convolution to directly extract local-global features from a sequence of raw frames, without the prior knowledge of key frames. The transformer-style fully-connected convolution is proposed to extract local features while maintaining global receptive fields, and the graph-style channel correspondence convolution is introduced to model the correlations among feature patterns. Moreover, MER, optical flow estimation, and facial landmark detection are jointly trained by sharing the local-global features. The two latter tasks contribute to capturing facial subtle action information for MER, which can alleviate the impact of insufficient training data. Extensive experiments demonstrate that our framework (i) outperforms the state-of-the-art MER methods on CASME II, SAMM, and SMIC benchmarks, (ii) works well for optical flow estimation and facial landmark detection, and (iii) can capture facial subtle muscle actions in local regions associated with MEs. The code is available at this https URL.
Abstract (translated)
面部微表情识别（MER）是一个具有挑战性的问题，由于短暂和细微的微表情动作。大多数现有方法依赖于手工设计特征、关键帧如起始点、顶峰点及结束点，或受制于小规模且低多样性数据集的深度网络。在本文中，我们提出了一种结合了变压器、图卷积以及普通卷积优势的端到端微动作感知深度学习框架。 特别地，我们提出了一种新颖的F5C（全连接卷积与通道对应卷积组成）模块，可以直接从一系列原始帧序列提取局部-全局特征，而无需事先知道关键帧的信息。提出的变压器风格的全连接卷积旨在提取局部特征的同时保持全局感受野，图样式通道对应卷积则被引入以建模特征模式之间的相关性。 此外，MER、光流估计和面部标志点检测通过共享局部-全局特征进行联合训练。后两项任务有助于捕捉对微表情识别有用的细微面部动作信息，从而缓解因训练数据不足造成的影响。 广泛的实验表明，我们的框架（i）在CASME II、SAMM 和 SMIC 评估基准上超越了最先进的MER方法，（ii）对于光流估计和面部标志点检测表现出良好的性能，并且（iii）能够捕捉与微表情相关的局部区域内的细微肌肉动作。代码可在[此处](https://this https URL)获取。
URL
https://arxiv.org/abs/2506.14511
PDF
-
OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis
2025-06-03 13:56:10Jiewen Hu, Leena Mathur, Paul Pu Liang, Louis-Philippe MorencyAbstract
In recent years, there has been increasing interest in automatic facial behavior analysis systems from computing communities such as vision, multimodal interaction, robotics, and affective computing. Building upon the widespread utility of prior open-source facial analysis systems, we introduce OpenFace 3.0, an open-source toolkit capable of facial landmark detection, facial action unit detection, eye-gaze estimation, and facial emotion recognition. OpenFace 3.0 contributes a lightweight unified model for facial analysis, trained with a multi-task architecture across diverse populations, head poses, lighting conditions, video resolutions, and facial analysis tasks. By leveraging the benefits of parameter sharing through a unified model and training paradigm, OpenFace 3.0 exhibits improvements in prediction performance, inference speed, and memory efficiency over similar toolkits and rivals state-of-the-art models. OpenFace 3.0 can be installed and run with a single line of code and operate in real-time without specialized hardware. OpenFace 3.0 code for training models and running the system is freely available for research purposes and supports contributions from the community.
Abstract (translated)
近年来，计算领域的视觉、多模态交互、机器人和情感计算社区对自动面部行为分析系统产生了越来越浓厚的兴趣。基于先前开源的面部分析系统的广泛应用，我们推出了OpenFace 3.0，这是一个能够进行面部标志点检测、面部动作单元检测、眼动估计以及面部情绪识别的开源工具包。 OpenFace 3.0 提供了一种轻量级的统一模型，用于面部分析，并且该模型通过多样化的人群样本、头部姿态、光照条件、视频分辨率和不同的面部分析任务进行了跨任务架构的训练。通过利用统一模型及训练范式中的参数共享带来的优势，OpenFace 3.0 在预测性能、推理速度以及内存效率方面超越了类似的工具包，并且在与现有先进模型的比较中表现出色。 OpenFace 3.0 可以用单行代码安装并运行，并能在没有专门硬件的情况下实时操作。用于训练模型和运行系统的 OpenFace 3.0 代码免费提供给研究目的使用，并支持社区贡献。
URL
https://arxiv.org/abs/2506.02891
PDF
-
Sheep Facial Pain Assessment Under Weighted Graph Neural Networks
2025-06-02 09:24:09Alam Noor, Luis Almeida, Mohamed Daoudi, Kai Li, Eduardo TovarAbstract
Accurately recognizing and assessing pain in sheep is key to discern animal health and mitigating harmful situations. However, such accuracy is limited by the ability to manage automatic monitoring of pain in those animals. Facial expression scoring is a widely used and useful method to evaluate pain in both humans and other living beings. Researchers also analyzed the facial expressions of sheep to assess their health state and concluded that facial landmark detection and pain level prediction are essential. For this purpose, we propose a novel weighted graph neural network (WGNN) model to link sheep's detected facial landmarks and define pain levels. Furthermore, we propose a new sheep facial landmarks dataset that adheres to the parameters of the Sheep Facial Expression Scale (SPFES). Currently, there is no comprehensive performance benchmark that specifically evaluates the use of graph neural networks (GNNs) on sheep facial landmark data to detect and measure pain levels. The YOLOv8n detector architecture achieves a mean average precision (mAP) of 59.30% with the sheep facial landmarks dataset, among seven other detection models. The WGNN framework has an accuracy of 92.71% for tracking multiple facial parts expressions with the YOLOv8n lightweight on-board device deployment-capable model.
Abstract (translated)
准确地识别和评估羊的疼痛对于判断动物健康状况以及减轻有害情况至关重要。然而，这种准确性受限于自动监测这些动物疼痛的能力。面部表情评分是一种广泛使用且有效的评估人类及其他生物疼痛的方法。研究人员还分析了羊的面部表情来评估其健康状态，并得出结论：面部特征检测及疼痛水平预测是至关重要的。为此，我们提出了一种新颖的加权图神经网络（WGNN）模型，用于连接所检测到的羊面部特征并定义疼痛级别。此外，我们提出了一个新的羊面部标志数据集，该数据集遵循《羊面部表情量表》(Sheep Pain Facial Expression Scale, SPFES) 的参数标准。 目前还没有一个全面的性能基准专门评估图神经网络（GNNs）在检测和测量羊面部特征数据疼痛水平方面的应用。YOLOv8n检测器架构在包含七个其他检测模型在内的羊面部标志数据集中，达到了平均精度均值 (mAP) 为59.30% 的成绩。使用 YOLOv8n 轻量级车载设备部署模型时，WGNN 框架对于跟踪多个面部部分表情的准确率为92.71%。
URL
https://arxiv.org/abs/2506.01468
PDF
-
FaceEditTalker: Interactive Talking Head Generation with Facial Attribute Editing
2025-05-28 09:04:00Guanwen Feng, Zhiyuan Ma, Yunan Li, Junwei Jing, Jiahao Yang, Qiguang MiaoAbstract
Recent advances in audio-driven talking head generation have achieved impressive results in lip synchronization and emotional expression. However, they largely overlook the crucial task of facial attribute editing. This capability is crucial for achieving deep personalization and expanding the range of practical applications, including user-tailored digital avatars, engaging online education content, and brand-specific digital customer service. In these key domains, the flexible adjustment of visual attributes-such as hairstyle, accessories, and subtle facial features is essential for aligning with user preferences, reflecting diverse brand identities, and adapting to varying contextual demands. In this paper, we present FaceEditTalker, a unified framework that enables controllable facial attribute manipulation while generating high-quality, audio-synchronized talking head videos. Our method consists of two key components: an image feature space editing module, which extracts semantic and detail features and allows flexible control over attributes like expression, hairstyle, and accessories; and an audio-driven video generation module, which fuses these edited features with audio-guided facial landmarks to drive a diffusion-based generator. This design ensures temporal coherence, visual fidelity, and identity preservation across frames. Extensive experiments on public datasets demonstrate that our method outperforms state-of-the-art approaches in lip-sync accuracy, video quality, and attribute controllability. Project page: this https URL
Abstract (translated)
近期在音频驱动的人脸生成领域的进展已经取得了唇部同步和情感表达方面的显著成果。然而，这些进步很大程度上忽略了关键的面部属性编辑任务。这一能力对于实现深度个性化以及扩大实际应用范围至关重要，包括用户定制的数字头像、互动性强的在线教育内容及品牌特定的数字化客户服务等。在这些重要领域中，能够灵活调整视觉属性（如发型、配饰和微妙的脸部特征）以符合用户的偏好、反映多样的品牌形象并适应不同的上下文需求是必不可少的。 在此论文中，我们提出了FaceEditTalker框架，该框架能够在生成高质量的音频同步人脸视频的同时实现可控制的面部属性编辑。我们的方法包含两个关键组件：一个图像特性空间编辑模块，它提取语义和细节特征，并允许对诸如表情、发型和配饰等属性进行灵活控制；以及一个基于音频驱动的视频生成模块，该模块将这些经过修改的特征与音频引导的人脸地标融合起来以驱动扩散型生成器。这种设计确保了帧间的时间一致性、视觉真实性和身份保持。 在公共数据集上的广泛实验表明，我们的方法在唇部同步准确性、视频质量和属性可控制性方面均优于当前最先进方法的表现。项目页面：[此链接](this https URL)
URL
https://arxiv.org/abs/2505.22141
PDF
-
LAMM-ViT: AI Face Detection via Layer-Aware Modulation of Region-Guided Attention
2025-05-12 16:42:19Jiangling Zhang, Weijie Zhu, Jirui Huang, Yaxiong ChenAbstract
Detecting AI-synthetic faces presents a critical challenge: it is hard to capture consistent structural relationships between facial regions across diverse generation techniques. Current methods, which focus on specific artifacts rather than fundamental inconsistencies, often fail when confronted with novel generative models. To address this limitation, we introduce Layer-aware Mask Modulation Vision Transformer (LAMM-ViT), a Vision Transformer designed for robust facial forgery detection. This model integrates distinct Region-Guided Multi-Head Attention (RG-MHA) and Layer-aware Mask Modulation (LAMM) components within each layer. RG-MHA utilizes facial landmarks to create regional attention masks, guiding the model to scrutinize architectural inconsistencies across different facial areas. Crucially, the separate LAMM module dynamically generates layer-specific parameters, including mask weights and gating values, based on network context. These parameters then modulate the behavior of RG-MHA, enabling adaptive adjustment of regional focus across network depths. This architecture facilitates the capture of subtle, hierarchical forgery cues ubiquitous among diverse generation techniques, such as GANs and Diffusion Models. In cross-model generalization tests, LAMM-ViT demonstrates superior performance, achieving 94.09% mean ACC (a +5.45% improvement over SoTA) and 98.62% mean AP (a +3.09% improvement). These results demonstrate LAMM-ViT's exceptional ability to generalize and its potential for reliable deployment against evolving synthetic media threats.
Abstract (translated)
检测AI生成的人脸提出了一个关键挑战：很难捕捉到不同生成技术之间面部区域间的一致结构性关系。当前的方法侧重于特定的伪影，而不是基本的不一致现象，在面对新型生成模型时往往失败。为了解决这一局限性，我们引入了层感知掩码调制视觉变换器（LAMM-ViT），这是一种专为鲁棒的人脸伪造检测设计的视觉变换器模型。该模型在每一层中集成了区域引导多头注意力（RG-MHA）和层感知掩码调制（LAMM）组件。 RG-MHA利用面部地标来创建区域注意图，引导模型审查不同面部区域间的架构不一致性。至关重要的是，单独的LAMM模块基于网络上下文动态生成特定于每一层的参数，包括掩码权重和门控值。这些参数随后调整RG-MHA的行为，使模型能够在网络深度上适应性地调节区域关注点。这种架构便于捕捉到不同生成技术（如GAN和扩散模型）中普遍存在但细微且层级化的伪造线索。 在跨模型泛化测试中，LAMM-ViT表现出卓越的性能，实现了平均准确率（ACC）94.09%（比现有最佳方法高出5.45%），以及平均精确召回率（AP）98.62%（比现有最佳方法高3.09%）。这些结果证明了LAMM-ViT具备出色的泛化能力及其在应对不断演化的合成媒体威胁方面的可靠部署潜力。
URL
https://arxiv.org/abs/2505.07734
PDF
-
Semantic Style Transfer for Enhancing Animal Facial Landmark Detection
2025-05-08 20:48:15Anadil Hussein, Anna Zamansky, George MartvelAbstract
Neural Style Transfer (NST) is a technique for applying the visual characteristics of one image onto another while preserving structural content. Traditionally used for artistic transformations, NST has recently been adapted, e.g., for domain adaptation and data augmentation. This study investigates the use of this technique for enhancing animal facial landmark detectors training. As a case study, we use a recently introduced Ensemble Landmark Detector for 48 anatomical cat facial landmarks and the CatFLW dataset it was trained on, making three main contributions. First, we demonstrate that applying style transfer to cropped facial images rather than full-body images enhances structural consistency, improving the quality of generated images. Secondly, replacing training images with style-transferred versions raised challenges of annotation misalignment, but Supervised Style Transfer (SST) - which selects style sources based on landmark accuracy - retained up to 98% of baseline accuracy. Finally, augmenting the dataset with style-transferred images further improved robustness, outperforming traditional augmentation methods. These findings establish semantic style transfer as an effective augmentation strategy for enhancing the performance of facial landmark detection models for animals and beyond. While this study focuses on cat facial landmarks, the proposed method can be generalized to other species and landmark detection models.
Abstract (translated)
神经风格迁移（NST）是一种将一幅图像的视觉特征应用于另一幅图象以保留结构内容的技术。这种技术传统上用于艺术转换，但最近已被改编用于领域适应和数据增强等领域。本研究探讨了该技术在提升动物面部标志检测器训练效果中的应用。作为案例研究，我们使用了一种为48个解剖学猫面部标志设计的集合标志探测器以及其基于CatFLW数据集进行的训练，并做出三项主要贡献。 首先，研究表明对裁剪后的面部图像而非全身图像应用风格迁移可以增强结构一致性，从而提高生成图像的质量。其次，在用经过风格转换的图像替换训练图像时出现了注释不一致的问题，但通过监督式风格传输（SST）——一种基于标志准确性选择样式来源的方法——能够保留高达98%的基础准确率。最后，将数据集与经过风格迁移的图像进行扩充进一步提升了模型的鲁棒性，并超越了传统的增强方法。 这些发现确立了语义风格迁移作为一种有效的增强策略，可以用于提升动物面部标志检测模型及其他相关领域的性能表现。虽然本研究重点是猫面部标志，但提出的方法也可以推广到其他物种和标志检测模型上。
URL
https://arxiv.org/abs/2505.05640
PDF
-
Visual and textual prompts for enhancing emotion recognition in video
2025-04-24 03:26:30Zhifeng Wang, Qixuan Zhang, Peter Zhang, Wenjia Niu, Kaihao Zhang, Ramesh Sankaranarayana, Sabrina Caldwell, Tom GedeonAbstract
Vision Large Language Models (VLLMs) exhibit promising potential for multi-modal understanding, yet their application to video-based emotion recognition remains limited by insufficient spatial and contextual awareness. Traditional approaches, which prioritize isolated facial features, often neglect critical non-verbal cues such as body language, environmental context, and social interactions, leading to reduced robustness in real-world scenarios. To address this gap, we propose Set-of-Vision-Text Prompting (SoVTP), a novel framework that enhances zero-shot emotion recognition by integrating spatial annotations (e.g., bounding boxes, facial landmarks), physiological signals (facial action units), and contextual cues (body posture, scene dynamics, others' emotions) into a unified prompting strategy. SoVTP preserves holistic scene information while enabling fine-grained analysis of facial muscle movements and interpersonal dynamics. Extensive experiments show that SoVTP achieves substantial improvements over existing visual prompting methods, demonstrating its effectiveness in enhancing VLLMs' video emotion recognition capabilities.
Abstract (translated)
视觉大型语言模型（VLLM）在多模态理解方面展现出巨大潜力，但它们在基于视频的情绪识别中的应用仍然受限于空间和上下文感知不足。传统方法侧重于孤立的面部特征，往往忽视了诸如肢体语言、环境背景和社会互动等关键非言语线索，导致其实用性在现实场景中大打折扣。为解决这一问题，我们提出了集合视觉-文本提示框架（SoVTP），这是一种新型框架，通过整合空间标注（如边界框、面部地标）、生理信号（面部动作单位）和上下文线索（身体姿态、场景动态以及他人的情绪），采用统一的提示策略来增强零样本情绪识别能力。SoVTP能够保留整体场景信息，并支持对面部肌肉运动和人际互动进行精细分析。广泛的实验表明，相较于现有的视觉提示方法，SoVTP实现了显著提升，在增强VLLM视频情感识别功能方面展现了有效性。
URL
https://arxiv.org/abs/2504.17224
PDF
