Abstract
Human perception integrates multiple modalities, such as vision, hearing, and language, into a unified understanding of the surrounding reality. While recent multimodal models have achieved significant progress by aligning pairs of modalities via contrastive learning, their solutions are unsuitable when scaling to multiple modalities. These models typically align each modality to a designated anchor without ensuring the alignment of all modalities with each other, leading to suboptimal performance in tasks requiring a joint understanding of multiple modalities. In this paper, we structurally rethink the pairwise conventional approach to multimodal learning and we present the novel Gramian Representation Alignment Measure (GRAM), which overcomes the above-mentioned limitations. GRAM learns and then aligns $n$ modalities directly in the higher-dimensional space in which modality embeddings lie by minimizing the Gramian volume of the $k$-dimensional parallelotope spanned by the modality vectors, ensuring the geometric alignment of all modalities simultaneously. GRAM can replace cosine similarity in any downstream method, holding for 2 to $n$ modality and providing more meaningful alignment with respect to previous similarity measures. The novel GRAM-based contrastive loss function enhances the alignment of multimodal models in the higher-dimensional embedding space, leading to new state-of-the-art performance in downstream tasks such as video-audio-text retrieval and audio-video classification. The project page, the code, and the pretrained models are available at this https URL.
Abstract (translated)
人类感知将多种模式（如视觉、听觉和语言）整合成对周围现实的统一理解。虽然最近的多模态模型通过对比学习对齐两种模式取得了显著进展，但这些解决方案在扩展到多种模式时并不适用。这些模型通常将每种模式与指定的锚点进行对齐，而没有确保所有模式之间的相互对齐，导致需要联合理解多种模式的任务性能不佳。本文重新思考了传统的多模态学习成对方法，并提出了新的Gramian表征对齐措施（GRAM），克服了上述限制。GRAM直接在模态嵌入存在的高维空间中学习并优化$n$种模式的对齐，通过最小化由模态向量张成的$k$维平行六面体的Gramian体积来确保所有模态的几何对齐。GRAM可以替代任何下游方法中的余弦相似性，适用于从2到$n$种模式，并提供了比先前相似度衡量更具有意义的对齐。基于GRAM的新对比损失函数增强了多模态模型在高维嵌入空间中的对齐效果，在视频-音频-文本检索和音频-视频分类等下游任务中达到了新的最先进的性能水平。项目页面、代码和预训练模型可以在提供的这个https URL地址获得。
URL
https://arxiv.org/abs/2412.11959
