Face_Detection
-
Investigating Adversarial Robustness against Preprocessing used in Blackbox Face Recognition
2025-10-20 05:19:35Roland Croft, Brian Du, Darcy Joseph, Sharath KumarAbstract
Face Recognition (FR) models have been shown to be vulnerable to adversarial examples that subtly alter benign facial images, exposing blind spots in these systems, as well as protecting user privacy. End-to-end FR systems first obtain preprocessed faces from diverse facial imagery prior to computing the similarity of the deep feature embeddings. Whilst face preprocessing is a critical component of FR systems, and hence adversarial attacks against them, we observe that this preprocessing is often overlooked in blackbox settings. Our study seeks to investigate the transferability of several out-of-the-box state-of-the-art adversarial attacks against FR when applied against different preprocessing techniques used in a blackbox setting. We observe that the choice of face detection model can degrade the attack success rate by up to 78%, whereas choice of interpolation method during downsampling has relatively minimal impacts. Furthermore, we find that the requirement for facial preprocessing even degrades attack strength in a whitebox setting, due to the unintended interaction of produced noise vectors against face detection models. Based on these findings, we propose a preprocessing-invariant method using input transformations that improves the transferability of the studied attacks by up to 27%. Our findings highlight the importance of preprocessing in FR systems, and the need for its consideration towards improving the adversarial generalisation of facial adversarial examples.
Abstract (translated)
面部识别（FR）模型已被证明容易受到微小改变的良性面部图像攻击，这些攻击揭示了系统的漏洞，并保护了用户隐私。端到端的面部识别系统在计算深度特征嵌入相似性之前，会从各种面部图像中获取预处理过的脸部。虽然面部预处理是面部识别系统的一个关键组成部分，因此也是对抗攻击的一部分，但我们观察到，在黑盒设置下，这一过程往往被忽视了。我们的研究旨在探讨几种现成的最先进的对抗攻击在应用于不同黑盒设置中的预处理技术时的效果转移性。 我们发现，选择不同的面部检测模型可以降低78%的攻击成功率，而降采样期间使用的插值方法的影响相对较小。此外，我们还发现对预处理的需求甚至会削弱白盒环境下的攻击力度，这是由于生成的噪声向量与面部检测模型之间的意外相互作用所致。 基于这些发现，我们提出了一种使用输入变换的方法来提高所研究攻击的效果转移性，这种方法不受预处理过程的影响，并将效果提高了多达27%。我们的研究表明，在面部识别系统中，预处理的重要性不容忽视，并且为了改善面部对抗样本的泛化能力，需要考虑这一因素。
URL
https://arxiv.org/abs/2510.17169
PDF
-
Extreme Amodal Face Detection
2025-10-08 09:22:03Changlin Song, Yunzhong Hou, Michael Randall Barnes, Rahul Shome, Dylan CampbellAbstract
Extreme amodal detection is the task of inferring the 2D location of objects that are not fully visible in the input image but are visible within an expanded field-of-view. This differs from amodal detection, where the object is partially visible within the input image, but is occluded. In this paper, we consider the sub-problem of face detection, since this class provides motivating applications involving safety and privacy, but do not tailor our method specifically to this class. Existing approaches rely on image sequences so that missing detections may be interpolated from surrounding frames or make use of generative models to sample possible completions. In contrast, we consider the single-image task and propose a more efficient, sample-free approach that makes use of the contextual cues from the image to infer the presence of unseen faces. We design a heatmap-based extreme amodal object detector that addresses the problem of efficiently predicting a lot (the out-of-frame region) from a little (the image) with a selective coarse-to-fine decoder. Our method establishes strong results for this new task, even outperforming less efficient generative approaches.
Abstract (translated)
极端模态检测（Extreme Amodal Detection）的任务是推断出输入图像中未完全可见但可以在扩展视野范围内看到的对象的二维位置。这与一般的模态检测不同，后者是指对象在输入图像中部分可见但被遮挡的情况。本文着重探讨了面部检测这一子问题，因为此类应用涉及安全和隐私领域的重要场景，但我们并未将方法专门针对这个类别进行设计。现有的方法依赖于图序列，以便从周围的帧中插值缺失的检测或利用生成模型来采样可能的完成情况。相比之下，我们考虑单一图像的任务，并提出了一种更高效的无需样本的方法，该方法利用来自图像的情境线索来推断未见人脸的存在。 为了应对这个任务，我们设计了一个基于热图（heatmap）的极端模态对象检测器，它解决了从少量信息高效预测大量数据的问题（即预测超出画面范围的部分），通过选择性地采用粗到细的解码器。我们的方法在这一新任务上取得了显著成果，并且优于其他更不高效的生成式方法。
URL
https://arxiv.org/abs/2510.06791
PDF
-
A Real-Time Multi-Model Parametric Representation of Point Clouds
2025-09-18 09:23:51Yuan Gao, Wei DongAbstract
In recent years, parametric representations of point clouds have been widely applied in tasks such as memory-efficient mapping and multi-robot collaboration. Highly adaptive models, like spline surfaces or quadrics, are computationally expensive in detection or fitting. In contrast, real-time methods, such as Gaussian mixture models or planes, have low degrees of freedom, making high accuracy with few primitives difficult. To tackle this problem, a multi-model parametric representation with real-time surface detection and fitting is proposed. Specifically, the Gaussian mixture model is first employed to segment the point cloud into multiple clusters. Then, flat clusters are selected and merged into planes or curved surfaces. Planes can be easily fitted and delimited by a 2D voxel-based boundary description method. Surfaces with curvature are fitted by B-spline surfaces and the same boundary description method is employed. Through evaluations on multiple public datasets, the proposed surface detection exhibits greater robustness than the state-of-the-art approach, with 3.78 times improvement in efficiency. Meanwhile, this representation achieves a 2-fold gain in accuracy over Gaussian mixture models, operating at 36.4 fps on a low-power onboard computer.
Abstract (translated)
近年来，点云的参数表示在诸如记忆高效地图构建和多机器人协作等任务中得到了广泛应用。高度自适应模型（如样条曲面或二次曲面）在检测或拟合过程中计算成本较高。相比之下，实时方法（例如高斯混合模型或平面模型），虽然自由度较低，但有助于提高效率，不过用少数几个基本元素实现高精度较为困难。为解决这一问题，提出了一种结合了多种模型的参数表示法，实现了实时表面检测和拟合。 具体来说，首先采用高斯混合模型将点云分割成多个簇。然后选择平面簇并将其合并成平面或曲面。平面可以通过基于二维体素边界描述的方法轻松拟合和限定。对于带有曲率的表面，则使用B样条曲面进行拟合，并且应用相同边界的描述方法。 通过在多个公开数据集上的评估，所提出的表面检测方法比现有最佳方法显示出更强的鲁棒性，在效率方面提高了3.78倍。同时，该表示法在准确性上较高斯混合模型提升了两倍，在低功耗车载计算机上的运行速度达到每秒36.4帧。
URL
https://arxiv.org/abs/2509.14773
PDF
-
Bridging Thoughts and Words: Graph-Based Intent-Semantic Joint Learning for Fake News Detection
2025-09-01 17:59:18Zhengjia Wang, Qiang Sheng, Danding Wang, Beizhe Hu, Juan CaoAbstract
Fake news detection is an important and challenging task for defending online information integrity. Existing state-of-the-art approaches typically extract news semantic clues, such as writing patterns that include emotional words, stylistic features, etc. However, detectors tuned solely to such semantic clues can easily fall into surface detection patterns, which can shift rapidly in dynamic environments, leading to limited performance in the evolving news landscape. To address this issue, this paper investigates a novel perspective by incorporating news intent into fake news detection, bridging intents and semantics together. The core insight is that by considering news intents, one can deeply understand the inherent thoughts behind news deception, rather than the surface patterns within words alone. To achieve this goal, we propose Graph-based Intent-Semantic Joint Modeling (InSide) for fake news detection, which models deception clues from both semantic and intent signals via graph-based joint learning. Specifically, InSide reformulates news semantic and intent signals into heterogeneous graph structures, enabling long-range context interaction through entity guidance and capturing both holistic and implementation-level intent via coarse-to-fine intent modeling. To achieve better alignment between semantics and intents, we further develop a dynamic pathway-based graph alignment strategy for effective message passing and aggregation across these signals by establishing a common space. Extensive experiments on four benchmark datasets demonstrate the superiority of the proposed InSide compared to state-of-the-art methods.
Abstract (translated)
虚假新闻检测是一项重要且具有挑战性的任务，旨在保护在线信息的完整性。现有的最先进的方法通常会提取诸如包含情感词汇的写作模式和风格特征等新闻语义线索。然而，仅针对此类语义线索调优的检测器容易陷入表面检测模式，在动态环境中这些模式变化迅速，从而在不断演变的新闻景观中表现有限。为了解决这个问题，本文从一个新颖的角度出发，通过将新闻意图纳入虚假新闻检测来探讨这一问题，旨在连接意图和语义之间的联系。核心见解在于：考虑新闻意图可以深入了解其背后的内在思维，而不仅仅是表面的文字模式。 为了实现这一目标，我们提出了一种基于图的意图-语义联合建模方法（InSide），用于假新闻检测。该模型通过基于图的联合学习方式从语义和意图信号中提取欺骗线索。具体而言，InSide将新闻的语义和意图信号重新表述为异构图结构，这使长距离上下文交互成为可能，并且能够通过粗到细级别的意图建模捕捉全面及实施层面的意图。 为了更好地实现语义与意图之间的对齐，我们进一步开发了一种基于动态路径图对齐策略的方法，以有效进行跨信号的消息传递和聚合，在此过程中建立了一个公共空间。在四个基准数据集上的广泛实验证明了所提出的InSide方法优于现有最先进的方法。
URL
https://arxiv.org/abs/2509.01660
PDF
-
SIExVulTS: Sensitive Information Exposure Vulnerability Detection System using Transformer Models and Static Analysis
2025-08-26 23:23:35Kyler Katz, Sara Moshtari, Ibrahim Mujhid, Mehdi Mirakhorli, Derek GarciaAbstract
Sensitive Information Exposure (SIEx) vulnerabilities (CWE-200) remain a persistent and under-addressed threat across software systems, often leading to serious security breaches. Existing detection tools rarely target the diverse subcategories of CWE-200 or provide context-aware analysis of code-level data flows. Aims: This paper aims to present SIExVulTS, a novel vulnerability detection system that integrates transformer-based models with static analysis to identify and verify sensitive information exposure in Java applications. Method: SIExVulTS employs a three-stage architecture: (1) an Attack Surface Detection Engine that uses sentence embeddings to identify sensitive variables, strings, comments, and sinks; (2) an Exposure Analysis Engine that instantiates CodeQL queries aligned with the CWE-200 hierarchy; and (3) a Flow Verification Engine that leverages GraphCodeBERT to semantically validate source-to-sink flows. We evaluate SIExVulTS using three curated datasets, including real-world CVEs, a benchmark set of synthetic CWE-200 examples, and labeled flows from 31 open-source projects. Results: The Attack Surface Detection Engine achieved an average F1 score greater than 93\%, the Exposure Analysis Engine achieved an F1 score of 85.71\%, and the Flow Verification Engine increased precision from 22.61\% to 87.23\%. Moreover, SIExVulTS successfully uncovered six previously unknown CVEs in major Apache projects. Conclusions: The results demonstrate that SIExVulTS is effective and practical for improving software security against sensitive data exposure, addressing limitations of existing tools in detecting and verifying CWE-200 vulnerabilities.
Abstract (translated)
敏感信息泄露（SIEx）漏洞（CWE-200）仍然是软件系统中持续存在的、未充分解决的安全威胁，经常导致严重的安全漏洞。现有的检测工具很少针对CWE-200的各种子类别进行目标定位或提供代码级数据流的上下文感知分析。本论文旨在介绍一种名为SIExVulTS的新颖漏洞检测系统，该系统整合了基于Transformer的模型与静态分析技术，以识别和验证Java应用程序中的敏感信息泄露。 **方法：** SIExVulTS采用三阶段架构： 1. **攻击面检测引擎**：使用句子嵌入来识别敏感变量、字符串、注释及接收点。 2. **暴露分析引擎**：实例化与CWE-200层级相匹配的CodeQL查询。 3. **流验证引擎**：利用GraphCodeBERT进行源到接收点数据流的语义验证。 我们使用三个经过整理的数据集来评估SIExVulTS，包括现实世界的CVE、一组合成的CWE-200示例以及来自31个开源项目标记的数据流。 **结果：** 攻击面检测引擎实现了平均F1分数超过93%，暴露分析引擎达到了85.71%的F1分数，而流验证引擎将精度从22.61%提升到了87.23%。此外，SIExVulTS在主要Apache项目中成功发现了六个此前未知的CVE。 **结论：** 结果表明，SIExVulTS对于改进软件安全以防止敏感数据泄露非常有效且实用，解决了现有工具在检测和验证CWE-200漏洞方面的局限性。
URL
https://arxiv.org/abs/2508.19472
PDF
-
Two-Factor Authentication Smart Entryway Using Modified LBPH Algorithm
2025-08-19 08:28:40Zakiah Ayop, Wan Mohamad Hariz Bin Wan Mohamad Rosdi, Looi Wei Hua, Syarulnaziah Anawar, Nur Fadzilah OthmanAbstract
Face mask detection has become increasingly important recently, particularly during the COVID-19 pandemic. Many face detection models have been developed in smart entryways using IoT. However, there is a lack of IoT development on face mask detection. This paper proposes a two-factor authentication system for smart entryway access control using facial recognition and passcode verification and an automation process to alert the owner and activate the surveillance system when a stranger is detected and controls the system remotely via Telegram on a Raspberry Pi platform. The system employs the Local Binary Patterns Histograms for the full face recognition algorithm and modified LBPH algorithm for occluded face detection. On average, the system achieved an Accuracy of approximately 70%, a Precision of approximately 80%, and a Recall of approximately 83.26% across all tested users. The results indicate that the system is capable of conducting face recognition and mask detection, automating the operation of the remote control to register users, locking or unlocking the door, and notifying the owner. The sample participants highly accept it for future use in the user acceptance test.
Abstract (translated)
最近，面部口罩检测变得越来越重要，特别是在COVID-19大流行期间。许多智能入口处的面部识别模型已经使用物联网技术开发出来，然而，在面部口罩检测方面的物联网开发却相对不足。本文提出了一种用于智能入口门禁控制的双因素认证系统，该系统结合了面部识别和密码验证，并在检测到陌生人时通过自动化过程向所有者发出警报并激活监控系统，同时还可以通过Telegram平台远程控制树莓派上的整个系统运行。 该系统采用局部二值模式直方图（LBPH）算法进行全脸识别，并且使用修改后的局部二值模式算法来检测被遮挡的面部。经过测试，在所有用户中，平均准确率约为70%，精确度约为80%，召回率为约83.26%。 结果显示，该系统能够执行面部识别和口罩检测、远程控制操作自动化（包括注册用户、锁定或解锁门以及通知所有者）。在用户体验接受性测试中，参与者对该系统的未来使用表示高度认可。
URL
https://arxiv.org/abs/2508.13617
PDF
-
Backdoor Attacks on Deep Learning Face Detection
2025-08-01 13:29:26Quentin Le Roux, Yannick Teglia, Teddy Furon, Philippe Loubet-MoundiAbstract
Face Recognition Systems that operate in unconstrained environments capture images under varying conditions,such as inconsistent lighting, or diverse face poses. These challenges require including a Face Detection module that regresses bounding boxes and landmark coordinates for proper Face Alignment. This paper shows the effectiveness of Object Generation Attacks on Face Detection, dubbed Face Generation Attacks, and demonstrates for the first time a Landmark Shift Attack that backdoors the coordinate regression task performed by face detectors. We then offer mitigations against these vulnerabilities.
Abstract (translated)
在非约束环境中运行的面部识别系统会在不同条件下捕捉图像，例如光照不一致或不同的面部姿态。这些挑战需要包括一个面部检测模块来回归边界框和地标坐标以实现正确的面部对齐。本文展示了针对面部检测系统的物体生成攻击（称为面部生成攻击）的有效性，并首次演示了一种地标偏移攻击，这种攻击会篡改面部探测器执行的坐标回归任务。最后，我们提供了针对这些漏洞的缓解措施。
URL
https://arxiv.org/abs/2508.00620
PDF
-
The Impact of Image Resolution on Face Detection: A Comparative Analysis of MTCNN, YOLOv XI and YOLOv XII models
2025-07-31 08:41:33Ahmet Can \"Omerciko\u{g}lu (D\"uzce University, Department of Computer Engineering, D\"uzce, T\"urkiye), Mustafa Mansur Y\"on\"ug\"ul (D\"uzce University, Department of Computer Engineering, D\"uzce, T\"urkiye), Pakize Erdo\u{g}mu\c{s} (D\"uzce University, Department of Computer Engineering, D\"uzce, T\"urkiye)Abstract
Face detection is a crucial component in many AI-driven applications such as surveillance, biometric authentication, and human-computer interaction. However, real-world conditions like low-resolution imagery present significant challenges that degrade detection performance. In this study, we systematically investigate the impact of input resolution on the accuracy and robustness of three prominent deep learning-based face detectors: YOLOv11, YOLOv12, and MTCNN. Using the WIDER FACE dataset, we conduct extensive evaluations across multiple image resolutions (160x160, 320x320, and 640x640) and assess each model's performance using metrics such as precision, recall, mAP50, mAP50-95, and inference time. Results indicate that YOLOv11 outperforms YOLOv12 and MTCNN in terms of detection accuracy, especially at higher resolutions, while YOLOv12 exhibits slightly better recall. MTCNN, although competitive in landmark localization, lags in real-time inference speed. Our findings provide actionable insights for selecting resolution-aware face detection models suitable for varying operational constraints.
Abstract (translated)
面部检测是许多人工智能驱动应用（如监控、生物识别认证和人机交互）的关键组成部分。然而，现实条件下的低分辨率图像对检测性能构成了重大挑战。在这项研究中，我们系统地调查了输入分辨率对三种基于深度学习的面部检测器——YOLOv11、YOLOv12 和 MTCNN 的准确性和鲁棒性的影响。使用 WIDER FACE 数据集，在多个图像分辨率（160x160、320x320 和 640x640）下进行了广泛的评估，并采用包括精度、召回率、mAP50、mAP50-95 及推断时间等指标来评估每个模型的性能。结果表明，YOLOv11 在检测准确性方面优于 YOLOv12 和 MTCNN，特别是在较高分辨率下，而 YOLOv12 的召回率略高。MTCNN 尽管在地标定位上具有竞争力，但在实时推断速度方面落后。我们的发现为选择适合各种操作约束的分辨率感知面部检测模型提供了可操作的见解。
URL
https://arxiv.org/abs/2507.23341
PDF
-
Bi-Level Optimization for Self-Supervised AI-Generated Face Detection
2025-07-30 16:38:29Mian Zou, Nan Zhong, Baosheng Yu, Yibing Zhan, Kede MaAbstract
AI-generated face detectors trained via supervised learning typically rely on synthesized images from specific generators, limiting their generalization to emerging generative techniques. To overcome this limitation, we introduce a self-supervised method based on bi-level optimization. In the inner loop, we pretrain a vision encoder only on photographic face images using a set of linearly weighted pretext tasks: classification of categorical exchangeable image file format (EXIF) tags, ranking of ordinal EXIF tags, and detection of artificial face manipulations. The outer loop then optimizes the relative weights of these pretext tasks to enhance the coarse-grained detection of manipulated faces, serving as a proxy task for identifying AI-generated faces. In doing so, it aligns self-supervised learning more closely with the ultimate goal of AI-generated face detection. Once pretrained, the encoder remains fixed, and AI-generated faces are detected either as anomalies under a Gaussian mixture model fitted to photographic face features or by a lightweight two-layer perceptron serving as a binary classifier. Extensive experiments demonstrate that our detectors significantly outperform existing approaches in both one-class and binary classification settings, exhibiting strong generalization to unseen generators.
Abstract (translated)
AI生成的面部检测器通过监督学习训练时，通常依赖于特定生成器合成的图像，这限制了它们对新兴生成技术的泛化能力。为克服这一局限性，我们提出了一种基于双层优化的自监督方法。在内部循环中，我们仅使用一组线性加权预设任务（如EXIF标签分类、顺序EXIF标签排序以及人工面部操作检测）对摄影人脸图像进行视觉编码器的预训练。外部循环则通过优化这些预设任务的相对权重来增强粗粒度的人脸篡改检测能力，作为识别AI生成面孔的代理任务，从而更紧密地将自监督学习与AI生成面部检测的最终目标相结合。完成预训练后，编码器保持固定状态不变，然后可以通过高斯混合模型适应摄影人脸特征下的异常检测或使用轻量级两层感知机进行二元分类来识别AI生成的人脸。 广泛的实验表明，在单类和二元分类设置下，我们的探测器在未见过的生成器上表现出显著优于现有方法的泛化能力。
URL
https://arxiv.org/abs/2507.22824
PDF
-
Bias Analysis for Synthetic Face Detection: A Case Study of the Impact of Facial Attributes
2025-07-29 13:45:39Asmae Lamsaf, Lucia Cascone, Hugo Proen\c{c}a, Jo\~ao NevesAbstract
Bias analysis for synthetic face detection is bound to become a critical topic in the coming years. Although many detection models have been developed and several datasets have been released to reliably identify synthetic content, one crucial aspect has been largely overlooked: these models and training datasets can be biased, leading to failures in detection for certain demographic groups and raising significant social, legal, and ethical issues. In this work, we introduce an evaluation framework to contribute to the analysis of bias of synthetic face detectors with respect to several facial attributes. This framework exploits synthetic data generation, with evenly distributed attribute labels, for mitigating any skew in the data that could otherwise influence the outcomes of bias analysis. We build on the proposed framework to provide an extensive case study of the bias level of five state-of-the-art detectors in synthetic datasets with 25 controlled facial attributes. While the results confirm that, in general, synthetic face detectors are biased towards the presence/absence of specific facial attributes, our study also sheds light on the origins of the observed bias through the analysis of the correlations with the balancing of facial attributes in the training sets of the detectors, and the analysis of detectors activation maps in image pairs with controlled attribute modifications.
Abstract (translated)
合成面部检测中的偏见分析必将成为未来几年的一个关键议题。尽管已经开发了许多检测模型，并且发布了几个可靠地识别合成内容的数据集，但一个重要的方面被广泛忽视了：这些模型和训练数据集可能会存在偏差，导致某些人口群体的检测失败，并引发重大的社会、法律和伦理问题。在本项工作中，我们介绍了一个评估框架，旨在对五种最先进的合成面部探测器相对于多种面部属性进行偏见水平分析。该框架利用具有均匀分布属性标签的合成数据生成技术来缓解可能影响偏见分析结果的数据偏差。 基于提出的框架，我们在包含25个受控面部属性的合成数据集中进行了详尽案例研究，以评估五种最先进的检测模型的偏见程度。虽然结果显示，总的来说，合成面部探测器对特定面部特征的存在或缺失存在偏向性，但我们的研究还通过分析训练集中的面部属性平衡以及在具有控制属性修改的图像对中探测器激活图的变化，揭示了观察到的偏见的起源。 这项工作强调了需要更加关注和理解合成检测模型中存在的潜在偏差，并为未来的研究提供了重要的见解，以开发更公平、透明且可靠的技术。
URL
https://arxiv.org/abs/2507.19705
PDF
-
Evaluation of facial landmark localization performance in a surgical setting
2025-07-24 09:40:47Ines Frajtag, Marko \v{S}vaco, Filip \v{S}uligojAbstract
The use of robotics, computer vision, and their applications is becoming increasingly widespread in various fields, including medicine. Many face detection algorithms have found applications in neurosurgery, ophthalmology, and plastic surgery. A common challenge in using these algorithms is variable lighting conditions and the flexibility of detection positions to identify and precisely localize patients. The proposed experiment tests the MediaPipe algorithm for detecting facial landmarks in a controlled setting, using a robotic arm that automatically adjusts positions while the surgical light and the phantom remain in a fixed position. The results of this study demonstrate that the improved accuracy of facial landmark detection under surgical lighting significantly enhances the detection performance at larger yaw and pitch angles. The increase in standard deviation/dispersion occurs due to imprecise detection of selected facial landmarks. This analysis allows for a discussion on the potential integration of the MediaPipe algorithm into medical procedures.
Abstract (translated)
机器人技术、计算机视觉及其应用在包括医学在内的各个领域中越来越广泛。许多面部检测算法已经在神经外科、眼科和整形外科等领域找到了应用。使用这些算法的一个常见挑战是变化的照明条件以及为准确识别和定位患者而灵活调整检测位置的需求。本次实验在一个受控环境中测试了MediaPipe算法，用于在手术灯保持固定且模型头部静止的情况下，由机械臂自动调节不同角度来检测面部特征点。 该研究结果表明，在手术照明条件下改进后的面部特征点检测精度显著提高了偏航角（左右旋转）和俯仰角（上下倾斜）较大时的检测性能。标准偏差/离散度的增加是由于对选定面部特征点识别的不精确性所致。这项分析为MediaPipe算法在医疗程序中的潜在整合提供了讨论基础。
URL
https://arxiv.org/abs/2507.18248
PDF
-
Divisive Decisions: Improving Salience-Based Training for Generalization in Binary Classification Tasks
2025-07-22 20:17:08Jacob Piland, Chris Sweet, Adam CzajkaAbstract
Existing saliency-guided training approaches improve model generalization by incorporating a loss term that compares the model's class activation map (CAM) for a sample's true-class ({\it i.e.}, correct-label class) against a human reference saliency map. However, prior work has ignored the false-class CAM(s), that is the model's saliency obtained for incorrect-label class. We hypothesize that in binary tasks the true and false CAMs should diverge on the important classification features identified by humans (and reflected in human saliency maps). We use this hypothesis to motivate three new saliency-guided training methods incorporating both true- and false-class model's CAM into the training strategy and a novel post-hoc tool for identifying important features. We evaluate all introduced methods on several diverse binary close-set and open-set classification tasks, including synthetic face detection, biometric presentation attack detection, and classification of anomalies in chest X-ray scans, and find that the proposed methods improve generalization capabilities of deep learning models over traditional (true-class CAM only) saliency-guided training approaches. We offer source codes and model weights\footnote{GitHub repository link removed to preserve anonymity} to support reproducible research.
Abstract (translated)
现有的基于注意力引导的训练方法通过引入一个损失项来改进模型泛化能力，该损失项将模型对样本真实类别的类别激活图（CAM）与人类参考注意力图进行比较。然而，先前的工作忽略了错误类别（即错误标签类别）的CAM。我们假设在二元任务中，真类别和假类别的CAM应该在人类认为重要的分类特征上有所区分（这些特征体现在人类的注意力图中）。基于这一假设，我们提出了三种新的基于注意力引导训练方法，将模型的真类别和假类别的CAM同时纳入训练策略，并提出了一种新颖的事后工具来识别重要特征。我们在多个不同的二元闭集和开集分类任务上评估了所有引入的方法，包括合成人脸检测、生物计量攻击呈现检测以及胸部X光片异常分类，发现所提出的这些方法相比传统的仅基于真类别CAM的注意力引导训练方法能够提高深度学习模型的泛化能力。为了支持可重复研究，我们提供了源代码和模型权重\footnote{GitHub仓库链接已移除以保证匿名性}。
URL
https://arxiv.org/abs/2507.17000
PDF
-
QRetinex-Net: Quaternion-Valued Retinex Decomposition for Low-Level Computer Vision Applications
2025-07-22 15:17:24Sos Agaian, Vladimir FrantsAbstract
Images taken in low light often show color shift, low contrast, noise, and other artifacts that hurt computer-vision accuracy. Retinex theory addresses this by viewing an image S as the pixel-wise product of reflectance R and illumination I, mirroring the way people perceive stable object colors under changing light. The decomposition is ill-posed, and classic Retinex models have four key flaws: (i) they treat the red, green, and blue channels independently; (ii) they lack a neuroscientific model of color vision; (iii) they cannot perfectly rebuild the input image; and (iv) they do not explain human color constancy. We introduce the first Quaternion Retinex formulation, in which the scene is written as the Hamilton product of quaternion-valued reflectance and illumination. To gauge how well reflectance stays invariant, we propose the Reflectance Consistency Index. Tests on low-light crack inspection, face detection under varied lighting, and infrared-visible fusion show gains of 2-11 percent over leading methods, with better color fidelity, lower noise, and higher reflectance stability.
Abstract (translated)
在低光条件下拍摄的照片通常会出现色偏、对比度低、噪点和其他影响计算机视觉准确性的伪像。Retinex理论通过将图像S视为反射率R和照明I的逐像素乘积来解决这些问题，这种模型反映了人们在不同光照条件下感知稳定物体颜色的方式。然而，这种分解是不适定问题（ill-posed problem），经典的Retinex模型存在四个主要缺陷：(i) 它们独立处理红色、绿色和蓝色通道；(ii) 缺乏神经科学色彩视觉模型；(iii) 无法完美重建输入图像；以及(iv) 不能解释人类的色恒常性。 我们引入了首个四元数Retinex公式，在该公式中，场景被描述为四元数值反射率与照明的汉密尔顿积。为了评估反射率保持不变的程度，我们提出了反射一致性指标（Reflectance Consistency Index）。在低光裂纹检测、不同光照条件下的面部识别以及红外和可见光图像融合方面的测试显示，相较于领先的方法，我们的方法提高了2-11%的性能，在色彩保真度、噪声降低和反射率稳定性方面表现更优。
URL
https://arxiv.org/abs/2507.16683
PDF
-
Seeing Through Deepfakes: A Human-Inspired Framework for Multi-Face Detection
2025-07-20 03:53:52Juan Hu, Shaojing Fan, Terence SimAbstract
Multi-face deepfake videos are becoming increasingly prevalent, often appearing in natural social settings that challenge existing detection methods. Most current approaches excel at single-face detection but struggle in multi-face scenarios, due to a lack of awareness of crucial contextual cues. In this work, we develop a novel approach that leverages human cognition to analyze and defend against multi-face deepfake videos. Through a series of human studies, we systematically examine how people detect deepfake faces in social settings. Our quantitative analysis reveals four key cues humans rely on: scene-motion coherence, inter-face appearance compatibility, interpersonal gaze alignment, and face-body consistency. Guided by these insights, we introduce \textsf{HICOM}, a novel framework designed to detect every fake face in multi-face scenarios. Extensive experiments on benchmark datasets show that \textsf{HICOM} improves average accuracy by 3.3\% in in-dataset detection and 2.8\% under real-world perturbations. Moreover, it outperforms existing methods by 5.8\% on unseen datasets, demonstrating the generalization of human-inspired cues. \textsf{HICOM} further enhances interpretability by incorporating an LLM to provide human-readable explanations, making detection results more transparent and convincing. Our work sheds light on involving human factors to enhance defense against deepfakes.
Abstract (translated)
多面深度伪造视频正变得越来越普遍，通常出现在自然社交环境中，这使得现有的检测方法难以应对。大多数当前的方法在单人脸检测方面表现出色，但在处理多个人脸的场景时却遇到困难，因为它们缺乏对关键上下文线索的理解。在这项工作中，我们开发了一种新的方法，该方法利用人类认知来分析和防御多面深度伪造视频。通过一系列的人类研究，我们系统地检查了人们在社交环境中如何检测深度伪造人脸的关键线索。定量分析揭示了四个关键因素：场景运动一致性、人脸间外观兼容性、人际注视对齐以及面部与身体的一致性。 基于这些见解，我们引入了一个新的框架\textsf{HICOM}，旨在识别多面场景中每一个假脸。在基准数据集上的广泛实验表明，\textsf{HICOM}使内部分类检测的平均准确率提高了3.3%，在现实世界扰动下的准确性提高了2.8%。此外，在未见过的数据集中，它比现有方法高出5.8%，展示了人类启发式线索的一般化能力。通过整合LLM（大型语言模型）来提供可读性解释，\textsf{HICOM}进一步增强了检测结果的透明度和说服力。 我们的工作为利用人类因素增强对深度伪造防御的研究提供了新的思路。
URL
https://arxiv.org/abs/2507.14807
PDF
-
Secure and Efficient UAV-Based Face Detection via Homomorphic Encryption and Edge Computing
2025-07-14 02:07:08Nguyen Van Duc, Bui Duc Manh, Quang-Trung Luu, Dinh Thai Hoang, Van-Linh Nguyen, Diep N. NguyenAbstract
This paper aims to propose a novel machine learning (ML) approach incorporating Homomorphic Encryption (HE) to address privacy limitations in Unmanned Aerial Vehicles (UAV)-based face detection. Due to challenges related to distance, altitude, and face orientation, high-resolution imagery and sophisticated neural networks enable accurate face recognition in dynamic environments. However, privacy concerns arise from the extensive surveillance capabilities of UAVs. To resolve this issue, we propose a novel framework that integrates HE with advanced neural networks to secure facial data throughout the inference phase. This method ensures that facial data remains secure with minimal impact on detection accuracy. Specifically, the proposed system leverages the Cheon-Kim-Kim-Song (CKKS) scheme to perform computations directly on encrypted data, optimizing computational efficiency and security. Furthermore, we develop an effective data encoding method specifically designed to preprocess the raw facial data into CKKS form in a Single-Instruction-Multiple-Data (SIMD) manner. Building on this, we design a secure inference algorithm to compute on ciphertext without needing decryption. This approach not only protects data privacy during the processing of facial data but also enhances the efficiency of UAV-based face detection systems. Experimental results demonstrate that our method effectively balances privacy protection and detection performance, making it a viable solution for UAV-based secure face detection. Significantly, our approach (while maintaining data confidentially with HE encryption) can still achieve an accuracy of less than 1% compared to the benchmark without using encryption.
Abstract (translated)
本文旨在提出一种结合同态加密（HE）的新型机器学习（ML）方法，以解决无人机（UAV）面部检测中的隐私限制问题。由于距离、高度和面部方向的变化带来的挑战，高分辨率图像和复杂的神经网络能够在动态环境中实现准确的人脸识别。然而，UAV的强大监控能力引发了隐私担忧。为了解决这一问题，我们提出了一种新颖的框架，该框架将HE与高级神经网络相结合，在推理阶段保护面部数据的安全性。这种方法确保了面部数据在安全的前提下不会显著影响检测准确性。 具体而言，所提出的系统利用Cheon-Kim-Kim-Song（CKKS）方案直接对加密数据进行计算，从而优化了计算效率和安全性。此外，我们开发了一种有效的数据编码方法，该方法专门用于将原始面部数据预处理为适合CKKS的格式，并采用单指令多数据流（SIMD）方式进行处理。在此基础上，设计了一个安全推理算法，在不解密的情况下对加密后的数据进行计算。这种方法不仅在处理人脸数据时保护了数据隐私，还提高了基于UAV的人脸检测系统的效率。 实验结果表明，我们提出的方法能够有效平衡隐私保护与检测性能之间的关系，使其成为基于UAV的面部识别系统的一种可行解决方案。特别值得注意的是，在保持数据机密性的同时（通过HE加密），我们的方法相比不使用加密的标准基准仍能达到不到1%误差率的准确性。
URL
https://arxiv.org/abs/2507.09860
PDF
-
Survivability of Backdoor Attacks on Unconstrained Face Recognition Systems
2025-07-02 11:21:27Quentin Le Roux, Yannick Teglia, Teddy Furon, Philippe Loubet-Moundi, Eric BourbaoAbstract
The widespread use of deep learning face recognition raises several security concerns. Although prior works point at existing vulnerabilities, DNN backdoor attacks against real-life, unconstrained systems dealing with images captured in the wild remain a blind spot of the literature. This paper conducts the first system-level study of backdoors in deep learning-based face recognition systems. This paper yields four contributions by exploring the feasibility of DNN backdoors on these pipelines in a holistic fashion. We demonstrate for the first time two backdoor attacks on the face detection task: face generation and face landmark shift attacks. We then show that face feature extractors trained with large margin losses also fall victim to backdoor attacks. Combining our models, we then show using 20 possible pipeline configurations and 15 attack cases that a single backdoor enables an attacker to bypass the entire function of a system. Finally, we provide stakeholders with several best practices and countermeasures.
Abstract (translated)
深度学习面部识别的广泛应用引发了几项安全问题。尽管先前的研究指出了现有的漏洞，但针对实际环境中捕获图像的真实无约束系统进行DNN后门攻击仍然是文献中的盲点。本文首次对基于深度学习的面部识别系统的后门进行了系统级研究。本文通过全面探索这些管道中DNN后门的可能性，在四个方面做出了贡献。我们首次展示了在面部检测任务上的两个后门攻击：面部生成和面部关键点偏移攻击。接着，我们还表明使用大边距损失训练的面部特征提取器同样容易受到后门攻击的影响。结合我们的模型，我们通过20种可能的管道配置和15个攻击案例展示了一个单一的后门就足以使攻击者绕过整个系统的功能。最后，本文为利益相关者提供了几种最佳实践和对策。
URL
https://arxiv.org/abs/2507.01607
PDF
-
Speaking images. A novel framework for the automated self-description of artworks
2025-05-28 09:13:41Valentine Bernasconi, Gustavo MarfiaAbstract
Recent breakthroughs in generative AI have opened the door to new research perspectives in the domain of art and cultural heritage, where a large number of artifacts have been digitized. There is a need for innovation to ease the access and highlight the content of digital collections. Such innovations develop into creative explorations of the digital image in relation to its malleability and contemporary interpretation, in confrontation to the original historical object. Based on the concept of the autonomous image, we propose a new framework towards the production of self-explaining cultural artifacts using open-source large-language, face detection, text-to-speech and audio-to-animation models. The goal is to start from a digitized artwork and to automatically assemble a short video of the latter where the main character animates to explain its content. The whole process questions cultural biases encapsulated in large-language models, the potential of digital images and deepfakes of artworks for educational purposes, along with concerns of the field of art history regarding such creative diversions.
Abstract (translated)
近期在生成式人工智能领域的突破为艺术和文化遗产领域的新研究视角打开了大门，尤其是在大量文物已经数字化的情况下。为了简化数字收藏的访问并突出其内容，需要创新性的解决方案。这些创新激发了对可塑性和当代诠释下数字图像关系的创造性探索，并与原始历史对象形成了对比。 基于自主图像的概念，我们提出了一种新的框架，用于利用开源大型语言模型、面部检测技术、文字转语音和音频到动画转换模型自动生成具有自我解释性质的文化艺术品。我们的目标是从数字化的艺术作品出发，自动组装一个短片，在该短片中主要角色会动起来并解释其内容。 整个过程质疑了大型语言模型中的文化偏见，探讨了数字图像以及艺术作品的深度伪造技术在教育领域的潜在应用，并引发了艺术史领域对这类创意延伸的关注和担忧。
URL
https://arxiv.org/abs/2506.05368
PDF
-
Facial Attribute Based Text Guided Face Anonymization
2025-05-27 10:36:35Mustafa \.Izzet Mu\c{s}tu, Haz{\i}m Kemal EkenelAbstract
The increasing prevalence of computer vision applications necessitates handling vast amounts of visual data, often containing personal information. While this technology offers significant benefits, it should not compromise privacy. Data privacy regulations emphasize the need for individual consent for processing personal data, hindering researchers' ability to collect high-quality datasets containing the faces of the individuals. This paper presents a deep learning-based face anonymization pipeline to overcome this challenge. Unlike most of the existing methods, our method leverages recent advancements in diffusion-based inpainting models, eliminating the need for training Generative Adversarial Networks. The pipeline employs a three-stage approach: face detection with RetinaNet, feature extraction with VGG-Face, and realistic face generation using the state-of-the-art BrushNet diffusion model. BrushNet utilizes the entire image, face masks, and text prompts specifying desired facial attributes like age, ethnicity, gender, and expression. This enables the generation of natural-looking images with unrecognizable individuals, facilitating the creation of privacy-compliant datasets for computer vision research.
Abstract (translated)
计算机视觉应用程序的日益普及需要处理大量包含个人信息的视觉数据。虽然这项技术提供了许多好处，但不应以牺牲隐私为代价。数据隐私法规强调了在处理个人信息时获取个体同意的重要性，这限制了研究人员收集高质量的人脸数据集的能力。本文提出了一种基于深度学习的脸部匿名化管道，旨在克服这一挑战。 与大多数现有方法不同，我们的方法利用了基于扩散的图像修复模型的最新进展，从而无需训练生成对抗网络（GAN）。该管道采用三阶段方法：使用RetinaNet进行脸部检测、用VGG-Face提取特征以及利用最先进的BrushNet扩散模型生成逼真的面部图像。BrushNet利用整个图像、脸部掩码和文本提示（包括所需的面部属性如年龄、种族、性别和表情）来生成自然且无法识别个人身份的图像，从而促进计算机视觉研究中符合隐私合规的数据集的创建。
URL
https://arxiv.org/abs/2505.21002
PDF
-
MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labeling
2025-05-21 17:23:12Cheng Yifan, Zhang Ruoyi, Shi JiatongAbstract
Acquiring large-scale emotional speech data with strong consistency remains a challenge for speech synthesis. This paper presents MIKU-PAL, a fully automated multimodal pipeline for extracting high-consistency emotional speech from unlabeled video data. Leveraging face detection and tracking algorithms, we developed an automatic emotion analysis system using a multimodal large language model (MLLM). Our results demonstrate that MIKU-PAL can achieve human-level accuracy (68.5% on MELD) and superior consistency (0.93 Fleiss kappa score) while being much cheaper and faster than human annotation. With the high-quality, flexible, and consistent annotation from MIKU-PAL, we can annotate fine-grained speech emotion categories of up to 26 types, validated by human annotators with 83% rationality ratings. Based on our proposed system, we further released a fine-grained emotional speech dataset MIKU-EmoBench(131.2 hours) as a new benchmark for emotional text-to-speech and visual voice cloning.
Abstract (translated)
获取大规模且一致性较强的情感语音数据仍然是语音合成领域的一个挑战。本文介绍了一种名为MIKU-PAL的全自动多模态管道，用于从无标签视频数据中提取高一致性的感情语音。通过利用面部检测和跟踪算法，我们开发了一个基于多模态大型语言模型（MLLM）的自动情感分析系统。我们的结果表明，MIKU-PAL能够在MELD数据集上实现与人类标注相当的准确率（68.5%），并且具有更好的一致性（Fleiss Kappa评分为0.93），同时成本更低、速度更快。基于MIKU-PAL提供的高质量、灵活且一致的标注，我们能够为精细的情感语音类别进行标签标注，涵盖了多达26种类型，并得到了人类注释者的83%合理性评分的认可。 在此基础上，我们进一步发布了一个细粒度情感语音数据集MIKU-EmoBench（131.2小时），旨在成为情感文本到语音和视觉声音克隆的新基准。
URL
https://arxiv.org/abs/2505.15772
PDF
-
Colors Matter: AI-Driven Exploration of Human Feature Colors
2025-05-20 21:35:44Rama Alyoubi, Taif Alharbi, Albatul Alghamdi, Yara Alshehri, Elham AlghamdiAbstract
This study presents a robust framework that leverages advanced imaging techniques and machine learning for feature extraction and classification of key human attributes-namely skin tone, hair color, iris color, and vein-based undertones. The system employs a multi-stage pipeline involving face detection, region segmentation, and dominant color extraction to isolate and analyze these features. Techniques such as X-means clustering, alongside perceptually uniform distance metrics like Delta E (CIEDE2000), are applied within both LAB and HSV color spaces to enhance the accuracy of color differentiation. For classification, the dominant tones of the skin, hair, and iris are extracted and matched to a custom tone scale, while vein analysis from wrist images enables undertone classification into "Warm" or "Cool" based on LAB differences. Each module uses targeted segmentation and color space transformations to ensure perceptual precision. The system achieves up to 80% accuracy in tone classification using the Delta E-HSV method with Gaussian blur, demonstrating reliable performance across varied lighting and image conditions. This work highlights the potential of AI-powered color analysis and feature extraction for delivering inclusive, precise, and nuanced classification, supporting applications in beauty technology, digital personalization, and visual analytics.
Abstract (translated)
这项研究提出了一种利用先进成像技术和机器学习进行特征提取和分类的关键人类属性框架，这些关键属性包括肤色、发色、虹膜颜色以及基于静脉的底色调。系统采用多阶段管道，涉及面部检测、区域分割和主色调提取来分离并分析这些特征。 研究中采用了X-means聚类等技术，并在LAB和HSV色彩空间内应用感知一致的距离度量（如CIEDE2000中的Delta E），以提高颜色区分的准确性。对于分类而言，从皮肤、头发和虹膜中提取主要色调并与自定义色调等级匹配，同时手腕图像中的静脉分析基于LAB差异将底色分类为“暖”或“冷”。每个模块使用目标分割和色彩空间转换来确保感知精度。 该系统在使用Delta E-HSV方法并结合高斯模糊的情况下，在色调分类上实现了高达80%的准确率，展示了其在不同光照和图像条件下可靠的表现力。这项工作突显了AI驱动的颜色分析和特征提取在美容技术、数字个性化以及视觉分析中的潜力，可以实现包容性、精确且细致的分类支持。
URL
https://arxiv.org/abs/2505.14931
PDF
