Abstract
In multimedia applications such as films and video games, spatial audio techniques are widely employed to enhance user experiences by simulating 3D sound: transforming mono audio into binaural formats. However, this process is often complex and labor-intensive for sound designers, requiring precise synchronization of audio with the spatial positions of visual components. To address these challenges, we propose a visual-based spatial audio generation system - an automated system that integrates face detection YOLOv8 for object detection, monocular depth estimation, and spatial audio techniques. Notably, the system operates without requiring additional binaural dataset training. The proposed system is evaluated against existing Spatial Audio generation system using objective metrics. Experimental results demonstrate that our method significantly improves spatial consistency between audio and video, enhances speech quality, and performs robustly in multi-speaker scenarios. By streamlining the audio-visual alignment process, the proposed system enables sound engineers to achieve high-quality results efficiently, making it a valuable tool for professionals in multimedia production.
Abstract (translated)
在多媒体应用（如电影和视频游戏）中，空间音频技术被广泛用于通过模拟三维声音来增强用户体验：将单声道音频转换为双耳格式。然而，这个过程对于音效设计师来说往往复杂且劳动密集型，需要精确地将音频与视觉组件的空间位置同步。为了应对这些挑战，我们提出了一种基于视觉的空间音频生成系统——一个集成了对象检测（使用YOLOv8进行面部检测）、单目深度估计和空间音频技术的自动化系统。值得注意的是，该系统在不需额外双耳数据集训练的情况下运行。 我们的提议系统通过客观指标与现有的空间音频生成系统进行了比较评估。实验结果表明，我们提出的方法显著提高了音频和视频之间的空间一致性，增强了语音质量，并且在多说话人场景中表现出良好的鲁棒性。通过简化音视同步的过程，所提出的系统使声音工程师能够高效地获得高质量的结果，成为多媒体制作专业人士的宝贵工具。
URL
https://arxiv.org/abs/2502.07538
