Abstract
The recent realistic creation and dissemination of so-called deepfakes poses a serious threat to social life, civil rest, and law. Celebrity defaming, election manipulation, and deepfakes as evidence in court of law are few potential consequences of deepfakes. The availability of open source trained models based on modern frameworks such as PyTorch or TensorFlow, video manipulations Apps such as FaceApp and REFACE, and economical computing infrastructure has easen the creation of deepfakes. Most of the existing detectors focus on detecting either face-swap, lip-sync, or puppet master deepfakes, but a unified framework to detect all three types of deepfakes is hardly explored. This paper presents a unified framework that exploits the power of proposed feature fusion of hybrid facial landmarks and our novel heart rate features for detection of all types of deepfakes. We propose novel heart rate features and fused them with the facial landmark features to better extract the facial artifacts of fake videos and natural variations available in the original videos. We used these features to train a light-weight XGBoost to classify between the deepfake and bonafide videos. We evaluated the performance of our framework on the world leaders dataset (WLDR) that contains all types of deepfakes. Experimental results illustrate that the proposed framework offers superior detection performance over the comparative deepfakes detection methods. Performance comparison of our framework against the LSTM-FCN, a candidate of deep learning model, shows that proposed model achieves similar results, however, it is more interpretable.
Abstract (translated)
最近，所谓深度伪造（deepfake）的现实创作和传播对社会生活、民众安宁以及法律秩序构成了严重威胁。深度伪造可能带来的后果包括名人诽谤、选举操纵以及在法庭上使用作为证据的深度伪造视频等。现代框架如PyTorch或TensorFlow的开源训练模型、FaceApp 和 REFACE 等视频编辑应用程序，再加上经济实惠的计算基础设施，使得创建深度伪造变得更加容易。 现有的大部分检测工具主要针对脸部替换、嘴唇同步或者操纵者的深度伪造进行识别，但一种能够同时识别这三种类型深度伪造的统一框架却鲜有研究。本文提出了一种利用混合面部特征和我们创新的心率特征融合技术来检测所有类型的深度伪造的统一框架。我们提出了新颖的心率特征，并将其与面部地标特征相结合，以便更好地提取假视频中的面部伪迹以及原始视频中自然的变化。 我们使用这些特征训练了一个轻量级的XGBoost模型，用于区分深度伪造视频和真实视频。我们在包含各种类型深度伪造的世界领导人数据集（WLDR）上评估了该框架的性能。实验结果表明，所提出的框架在与比较方法进行深度伪造检测时具有优越的表现。我们还展示了我们的框架相对于LSTM-FCN这种深度学习模型候选者的性能对比，结果显示提议的方法可以获得类似的结果，但更具可解释性。
URL
https://arxiv.org/abs/2501.11927
