Abstract
Recent advances in interactive text-to-image retrieval (I-TIR) use diffusion models to bridge the modality gap between the textual information need and the images to be searched, resulting in increased effectiveness. However, existing frameworks fuse multi-modal views of user feedback by simple embedding addition. In this work, we show that this static and undifferentiated fusion indiscriminately incorporates generative noise produced by the diffusion model, leading to performance degradation for up to 55.62% samples. We further propose ADaFuSE (Adaptive Diffusion-Text Fusion with Semantic-aware Experts), a lightweight fusion model designed to align and calibrate multi-modal views for diffusion-augmented I-TIR, which can be plugged into existing frameworks without modifying the backbone encoder. Specifically, we introduce a dual-branch fusion mechanism that employs an adaptive gating branch to dynamically balance modality reliability, alongside a semantic-aware mixture-of-experts branch to capture fine-grained cross-modal nuances. Via thorough evaluation over four standard I-TIR benchmarks, ADaFuSE achieves state-of-the-art performance, surpassing DAR by up to 3.49% in Hits@10 with only a 5.29% parameter increase, while exhibiting stronger robustness to noisy and longer interactive queries. These results show that generative augmentation coupled with principled fusion provides a simple, generalizable alternative to fine-tuning for interactive retrieval.
Abstract (translated)
近期交互式文本到图像检索（I-TIR）领域的进展利用扩散模型弥合文本信息需求与待检索图像之间的模态鸿沟，从而提升了检索效果。然而，现有框架通过简单的嵌入相加来融合用户反馈的多模态视图。本研究指出，这种静态且无差别的融合方式会不加分辨地纳入扩散模型产生的生成噪声，导致高达55.62%样本的性能下降。我们进一步提出ADaFuSE（基于语义感知专家的自适应扩散-文本融合），这是一个轻量级融合模型，旨在对齐和校准扩散增强型I-TIR的多模态视图，且无需修改主干编码器即可嵌入现有框架。具体而言，我们设计了双分支融合机制：自适应门控分支动态平衡模态可靠性，语义感知混合专家分支捕捉细粒度跨模态差异。在四个标准I-TIR基准上的全面评估表明，ADaFuSE达到最先进性能，在Hits@10指标上较DAR最高提升3.49%，而参数量仅增加5.29%，同时对噪声和长交互查询展现出更强鲁棒性。这些结果证明，生成增强结合原则性融合为交互式检索提供了一种无需微调的简单、可泛化方案。
URL
https://arxiv.org/abs/2603.21886
