Abstract
How does audio describe the world around us? In this work, we propose a method for generating images of visual scenes from diverse in-the-wild sounds. This cross-modal generation task is challenging due to the significant information gap between auditory and visual signals. We address this challenge by designing a model that aligns audio-visual modalities by enriching audio features with visual information and translating them into the visual latent space. These features are then fed into the pre-trained image generator to produce images. To enhance image quality, we use sound source localization to select audio-visual pairs with strong cross-modal correlations. Our method achieves substantially better results on the VEGAS and VGGSound datasets compared to previous work and demonstrates control over the generation process through simple manipulations to the input waveform or latent space. Furthermore, we analyze the geometric properties of the learned embedding space and demonstrate that our learning approach effectively aligns audio-visual signals for cross-modal generation. Based on this analysis, we show that our method is agnostic to specific design choices, showing its generalizability by integrating various model architectures and different types of audio-visual data.
Abstract (translated)
音频是如何描述我们周围世界的？在这项工作中，我们提出了一种从各种野外声音中生成视觉场景图像的方法。这个跨模态生成任务具有挑战性，因为听觉和视觉信号之间存在显著的信息差距。我们通过设计一种模型来应对这一挑战，该模型通过丰富音频特征的视觉信息并将这些信息转换到视觉潜在空间中来对齐视听模式。然后将这些特征输入预训练的图像生成器以产生图像。为了提高图像质量，我们使用声源定位技术选择具有强跨模态相关性的视听配对数据。我们的方法在VEGAS和VGGSound数据集上取得了明显优于先前工作的结果，并且通过简单地操纵输入波形或潜在空间来展示对生成过程的控制能力。此外，我们分析了学习嵌入空间的几何属性并展示了我们的学习方法有效地实现了跨模态生成中的视听信号对齐。基于这一分析，我们表明我们的方法不依赖于特定的设计选择，在整合各种模型架构和不同类型的声音视觉数据时显示出其通用性。
URL
https://arxiv.org/abs/2412.06209
