Abstract
Vision-Language Models like CLIP are extensively used for inter-modal tasks which involve both visual and text modalities. However, when the individual modality encoders are applied to inherently intra-modal tasks like image-to-image retrieval, their performance suffers from the intra-modal misalignment. In this paper we study intra-modal misalignment in CLIP with a focus on the role of the projectors that map pre-projection image and text embeddings into the shared embedding space. By analyzing the form of the cosine similarity applied to projected features, and its interaction with the contrastive CLIP loss, we show that there is an inter-modal operator responsible for aligning the two modalities during training, and a second, intra-modal operator that only enforces intra-modal normalization but does nothing to promote intra-modal alignment. Via spectral analysis of the inter-modal operator, we identify an approximately isotropic subspace in which the two modalities are well-aligned, as well as anisotropic directions specific to each modality. We demonstrate that this aligned subspace can be directly obtained from the projector weights and that removing the anisotropic directions improves intra-modal alignment. Our experiments on intra-modal retrieval and classification benchmarks show that our training-free method reduces intra-modal misalignment, greatly lowers latency, and outperforms existing approaches across multiple pre-trained CLIP-like models. The code is publicly available at: this https URL.
Abstract (translated)
像CLIP这样的视觉-语言模型被广泛用于涉及视觉和文本模态的跨模态任务。然而，当单独的模态编码器应用于图像到图像检索等固有的模态内任务时，其性能会受到模态内错位的影响。本文研究了CLIP中的模态内错位问题，重点关注将预投影图像和文本嵌入映射到共享嵌入空间的投影器的作用。通过分析应用于投影特征的余弦相似度的形式及其与对比式CLIP损失的相互作用，我们发现在训练过程中存在一个负责对齐两个模态的跨模态算子，以及第二个仅强制执行模态内归一化但无助于促进模态内对齐的模态内算子。通过对跨模态算子进行谱分析，我们识别出一个近似各向同性的子空间，其中两个模态对齐良好，以及每个模态特有的各向异性方向。我们证明这个对齐子空间可以直接从投影器权重中获得，并且去除各向异性方向可以改善模态内对齐。我们在模态内检索和分类基准上的实验表明，我们这种无需训练的方法减少了模态内错位，显著降低了延迟，并在多个预训练的类CLIP模型上优于现有方法。代码已在以下网址公开提供：此https链接。
URL
https://arxiv.org/abs/2603.19862
