Abstract
Audio-driven talking face generation is a widely researched topic due to its high applicability. Reconstructing a talking face using audio significantly contributes to fields such as education, healthcare, online conversations, virtual assistants, and virtual reality. Early studies often focused solely on changing the mouth movements, which resulted in outcomes with limited practical applications. Recently, researchers have proposed a new approach of constructing the entire face, including face pose, neck, and shoulders. To achieve this, they need to generate through landmarks. However, creating stable landmarks that align well with the audio is a challenge. In this paper, we propose the KFusion of Dual-Domain model, a robust model that generates landmarks from audio. We separate the audio into two distinct domains to learn emotional information and facial context, then use a fusion mechanism based on the KAN model. Our model demonstrates high efficiency compared to recent models. This will lay the groundwork for the development of the audio-driven talking face generation problem in the future.
Abstract (translated)
音频驱动的对话式面部生成是一个广泛研究的话题，因为它的实际应用非常广泛。通过音频重构对话式面部在教育、医疗、在线对话、虚拟助手和虚拟现实等领域具有重要意义。早期的研究通常仅关注改变嘴部的运动，导致实际应用效果有限。近年来，研究人员提出了一种新的方法来构建整个面部，包括脸部姿势、脖子和肩膀。要实现这一点，他们需要通过地标进行生成。然而，创建与音频一致的对齐稳定的地标是一个挑战。在本文中，我们提出了KFusion双域模型，一种基于音频生成地标的有前途的模型。我们将音频分为两个不同的领域来学习情感信息和面部上下文，然后使用基于KAN模型的融合机制。我们的模型在最近模型的效率方面表现出优异的表现。这将为实现音频驱动对话式面部生成问题的未来发展奠定了基础。
URL
https://arxiv.org/abs/2409.05330
