Image_Retrieval
-
Bag of Bags: Adaptive Visual Vocabularies for Genizah Join Image Retrieval
2026-04-09 11:55:34Sharva Gogawale, Gal Grudka, Daria Vasyutinsky-Shapira, Omer Ventura, Berat Kurar-Barakat, Nachum DershowitzAbstract
A join is a set of manuscript fragments identified as originally emanating from the same manuscript. We study manuscript join retrieval: Given a query image of a fragment, retrieve other fragments originating from the same physical manuscript. We propose Bag of Bags (BoB), an image-level representation that replaces the global-level visual codebook of classical Bag of Words (BoW) with a fragment-specific vocabulary of local visual words. Our pipeline trains a sparse convolutional autoencoder on binarized fragment patches, encodes connected components from each page, clusters the resulting embeddings with per image $k$-means, and compares images using set to set distances between their local vocabularies. Evaluated on fragments from the Cairo Genizah, the best BoB variant (viz.\@ Chamfer) achieves Hit@1 of 0.78 and MRR of 0.84, compared to 0.74 and 0.80, respectively, for the strongest BoW baseline (BoW-RawPatches-$\chi^2$), a 6.1\% relative improvement in top-1 accuracy. We furthermore study a mass-weighted BoB-OT variant that incorporates cluster population into prototype matching and present a formal approximation guarantee bounding its deviation from full component-level optimal transport. A two-stage pipeline using a BoW shortlist followed by BoB-OT reranking provides a practical compromise between retrieval strength and computational cost, supporting applicability to larger manuscript collections.
Abstract (translated)
URL
https://arxiv.org/abs/2604.08138
PDF
-
SearchAD: Large-Scale Rare Image Retrieval Dataset for Autonomous Driving
2026-04-09 09:10:41Felix Embacher, Jonas Uhrig, Marius Cordts, Markus EnzweilerAbstract
Retrieving rare and safety-critical driving scenarios from large-scale datasets is essential for building robust autonomous driving (AD) systems. As dataset sizes continue to grow, the key challenge shifts from collecting more data to efficiently identifying the most relevant samples. We introduce SearchAD, a large-scale rare image retrieval dataset for AD containing over 423k frames drawn from 11 established datasets. SearchAD provides high-quality manual annotations of more than 513k bounding boxes covering 90 rare categories. It specifically targets the needle-in-a-haystack problem of locating extremely rare classes, with some appearing fewer than 50 times across the entire dataset. Unlike existing benchmarks, which focused on instance-level retrieval, SearchAD emphasizes semantic image retrieval with a well-defined data split, enabling text-to-image and image-to-image retrieval, few-shot learning, and fine-tuning of multi-modal retrieval models. Comprehensive evaluations show that text-based methods outperform image-based ones due to stronger inherent semantic grounding. While models directly aligning spatial visual features with language achieve the best zero-shot results, and our fine-tuning baseline significantly improves performance, absolute retrieval capabilities remain unsatisfactory. With a held-out test set on a public benchmark server, SearchAD establishes the first large-scale dataset for retrieval-driven data curation and long-tail perception research in AD: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2604.08008
PDF
-
Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering
2026-04-08 14:37:35Zhuohong Chen, Zhenxian Wu, Yunyao Yu, Hangrui Xu, Zirui Liao, Zhifang Liu, Xiangwen Deng, Pen Jiao, Haoqian WangAbstract
Knowledge-based visual question answering (KB-VQA) requires vision-language models to understand images and use external knowledge, especially for rare entities and long-tail facts. Most existing retrieval-augmented generation (RAG) methods adopt a fixed pipeline that sequentially retrieves information, filters it, and then produces an answer. Such a design makes it difficult to adapt to diverse question types. Moreover, it separates retrieval from reasoning, making it hard for the model to decide when to search, how to refine queries, or when to stop. As a result, the retrieved evidence is often poorly aligned with the question. To address these limitations, we reformulate KB-VQA as a search-agent problem and model the solving process as a multi-step decision-making procedure. At each step, the agent selects one of four actions-Answer, Image Retrieval, Text Retrieval, and Caption-based on its current information state. We further design an automated pipeline to collect multi-step trajectories that record the agent's reasoning process, tool usage, and intermediate decisions. These trajectories are then used as supervision for fine-tuning. Experiments on InfoSeek and E-VQA demonstrate that our method achieves state-of-the-art performance, consistently outperforming prior baselines and confirming the effectiveness of our framework.
Abstract (translated)
URL
https://arxiv.org/abs/2604.07146
PDF
-
WRF4CIR: Weight-Regularized Fine-Tuning Network for Composed Image Retrieval
2026-04-07 08:23:39Yizhuo Xu, Chaojian Yu, Yuanjie Shao, Tongliang Liu, Qinmu Peng, Xinge YouAbstract
Composed Image Retrieval (CIR) task aims to retrieve target images based on reference images and modification texts. Current CIR methods primarily rely on fine-tuning vision-language pre-trained models. However, we find that these approaches commonly suffer from severe overfitting, posing challenges for CIR with limited triplet data. To better understand this issue, we present a systematic study of overfitting in VLP-based CIR, revealing a significant and previously overlooked generalization gap across different models and datasets. Motivated by these findings, we introduce WRF4CIR, a Weight-Regularized Fine-tuning network for CIR. Specifically, during the fine-tuning process, we apply adversarial perturbations to the model weights for regularization, where these perturbations are generated in the opposite direction of gradient descent. Intuitively, WRF4CIR increases the difficulty of fitting the training data, which helps mitigate overfitting in CIR under limited triplet supervision. Extensive experiments on benchmark datasets demonstrate that WRF4CIR significantly narrows the generalization gap and achieves substantial improvements over existing methods.
Abstract (translated)
组合图像检索（CIR）任务旨在基于参考图像和修改文本检索目标图像。当前CIR方法主要依赖对视觉-语言预训练模型进行微调。然而，我们发现这些方法普遍存在严重的过拟合问题，给有限三元组数据下的CIR带来挑战。为深入理解此问题，我们对基于VLP的CIR中的过拟合进行了系统性研究，揭示了一个显著且此前被忽视的跨模型与数据集的泛化差距。受此发现启发，我们提出了WRF4CIR，一种用于CIR的权重正则化微调网络。具体而言，在微调过程中，我们通过对模型权重施加对抗性扰动进行正则化，这些扰动在梯度下降的反方向生成。直观上，WRF4CIR增加了拟合训练数据的难度，从而有助于缓解有限三元组监督下CIR的过拟合问题。在基准数据集上的大量实验表明，WRF4CIR显著缩小了泛化差距，并相较于现有方法取得了实质性提升。
URL
https://arxiv.org/abs/2604.05583
PDF
-
Pretrain-then-Adapt: Uncertainty-Aware Test-Time Adaptation for Text-based Person Search
2026-04-07 07:48:15Jiahao Zhang, Shaofei Huang, Yaxiong Wang, Zhedong ZhengAbstract
Text-based person search faces inherent limitations due to data scarcity, driven by stringent privacy constraints and the high cost of manual annotation. To mitigate this, existing methods usually rely on a Pretrain-then-Finetune paradigm, where models are first pretrained on synthetic person-caption data to establish cross-modal alignment, followed by fine-tuning on labeled real-world datasets. However, this paradigm lacks practicality in real-world deployment scenarios, where large-scale annotated target-domain data is typically inaccessible. In this work, we propose a new Pretrain-then-Adapt paradigm that eliminates reliance on extensive target-domain supervision through an offline test-time adaptation manner, enabling dynamic model adaptation using only unlabeled test data with minimal post-train time cost. To mitigate overconfidence with false positives of previous entropy-based test-time adaptation, we propose an Uncertainty-Aware Test-Time Adaptation (UATTA) framework, which introduces a bidirectional retrieval disagreement mechanism to estimate uncertainty, i.e., low uncertainty is assigned when an image-text pair ranks highly in both image-to-text and text-to-image retrieval, indicating high alignment; otherwise, high uncertainty is detected. This indicator drives offline test-time model recalibration without labels, effectively mitigating domain shift. We validate UATTA on four benchmarks, i.e., CUHK-PEDES, ICFG-PEDES, RSTPReid, and PAB, showing consistent improvements across both CLIP-based (one-stage) and XVLM-based (two-stage) frameworks. Ablation studies confirm that UATTA outperforms existing offline test-time adaptation strategies, establishing a new benchmark for label-efficient, deployable person search systems. Our code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.08598
PDF
-
Beyond Semantic Search: Towards Referential Anchoring in Composed Image Retrieval
2026-04-07 03:43:01Yuxin Yang, Yinan Zhou, Yuxin Chen, Ziqi Zhang, Zongyang Ma, Chunfeng Yuan, Bing Li, Jun Gao, Weiming HuAbstract
Composed Image Retrieval (CIR) has demonstrated significant potential by enabling flexible multimodal queries that combine a reference image and modification text. However, CIR inherently prioritizes semantic matching, struggling to reliably retrieve a user-specified instance across contexts. In practice, emphasizing concrete instance fidelity over broad semantics is often more consequential. In this work, we propose Object-Anchored Composed Image Retrieval (OACIR), a novel fine-grained retrieval task that mandates strict instance-level consistency. To advance research on this task, we construct OACIRR (OACIR on Real-world images), the first large-scale, multi-domain benchmark comprising over 160K quadruples and four challenging candidate galleries enriched with hard-negative instance distractors. Each quadruple augments the compositional query with a bounding box that visually anchors the object in the reference image, providing a precise and flexible way to ensure instance preservation. To address the OACIR task, we propose AdaFocal, a framework featuring a Context-Aware Attention Modulator that adaptively intensifies attention within the specified instance region, dynamically balancing focus between the anchored instance and the broader compositional context. Extensive experiments demonstrate that AdaFocal substantially outperforms existing compositional retrieval models, particularly in maintaining instance-level fidelity, thereby establishing a robust baseline for this challenging task while opening new directions for more flexible, instance-aware retrieval systems.
Abstract (translated)
URL
https://arxiv.org/abs/2604.05393
PDF
-
CraterBench-R: Instance-Level Crater Retrieval for Planetary Scale
2026-04-06 03:04:31Jichao Fang, Lei Zhang, Michael Phillips, Wei LuoAbstract
Impact craters are a cornerstone of planetary surface analysis. However, while most deep learning pipelines treat craters solely as a detection problem, critical scientific workflows such as catalog deduplication, cross-observation matching, and morphological analog discovery are inherently retrieval tasks. To address this, we formulate crater analysis as an instance-level image retrieval problem and introduce CraterBench-R, a curated benchmark featuring about 25,000 crater identities with multi-scale gallery views and manually verified queries spanning diverse scales and contexts. Our baseline evaluations across various architectures reveal that self-supervised Vision Transformers (ViTs), particularly those with in-domain pretraining, dominate the task, outperforming generic models with significantly more parameters. Furthermore, we demonstrate that retaining multiple ViT patch tokens for late-interaction matching dramatically improves accuracy over standard single-vector pooling. However, storing all tokens per image is operationally inefficient at a planetary scale. To close this efficiency gap, we propose instance-token aggregation, a scalable, training-free method that selects K seed tokens, assigns the remaining tokens to these seeds via cosine similarity, and aggregates each cluster into a single representative token. This approach yields substantial gains: at K=16, aggregation improves mAP by 17.9 points over raw token selection, and at K=64, it matches the accuracy of using all 196 tokens with significantly less storage. Finally, we demonstrate that a practical two-stage pipeline, with single-vector shortlisting followed by instance-token reranking, recovers 89-94% of the full late-interaction accuracy while searching only a small candidate set. The benchmark is publicly available at this http URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.06245
PDF
-
MELT: Improve Composed Image Retrieval via the Modification Frequentation-Rarity Balance Network
2026-03-31 05:52:58Guozhi Qiu, Zhiwei Chen, Zixu Li, Qinlei Huang, Zhiheng Fu, Xuemeng Song, Yupeng HuAbstract
Composed Image Retrieval (CIR) uses a reference image and a modification text as a query to retrieve a target image satisfying the requirement of ``modifying the reference image according to the text instructions''. However, existing CIR methods face two limitations: (1) frequency bias leading to ``Rare Sample Neglect'', and (2) susceptibility of similarity scores to interference from hard negative samples and noise. To address these limitations, we confront two key challenges: asymmetric rare semantic localization and robust similarity estimation under hard negative samples. To solve these challenges, we propose the Modification frEquentation-rarity baLance neTwork MELT. MELT assigns increased attention to rare modification semantics in multimodal contexts while applying diffusion-based denoising to hard negative samples with high similarity scores, enhancing multimodal fusion and matching. Extensive experiments on two CIR benchmarks validate the superior performance of MELT. Codes are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.29291
PDF
-
Beyond MACs: Hardware Efficient Architecture Design for Vision Backbones
2026-03-27 16:05:20Moritz Nottebaum, Matteo Dunnhofer, Christian MicheloniAbstract
Vision backbone networks play a central role in modern computer vision. Enhancing their efficiency directly benefits a wide range of downstream applications. To measure efficiency, many publications rely on MACs (Multiply Accumulate operations) as a predictor of execution time. In this paper, we experimentally demonstrate the shortcomings of such a metric, especially in the context of edge devices. By contrasting the MAC count and execution time of common architectural design elements, we identify key factors for efficient execution and provide insights to optimize backbone design. Based on these insights, we present LowFormer, a novel vision backbone family. LowFormer features a streamlined macro and micro design that includes Lowtention, a lightweight alternative to Multi-Head Self-Attention. Lowtention not only proves more efficient, but also enables superior results on ImageNet. Additionally, we present an edge GPU version of LowFormer, that can further improve upon its baseline's speed on edge GPU and desktop GPU. We demonstrate LowFormer's wide applicability by evaluating it on smaller image classification datasets, as well as adapting it to several downstream tasks, such as object detection, semantic segmentation, image retrieval, and visual object tracking. LowFormer models consistently achieve remarkable speed-ups across various hardware platforms compared to recent state-of-the-art backbones. Code and models are available at this https URL.
Abstract (translated)
视觉骨干网络在现代计算机视觉中扮演着核心角色。提升其效率可直接惠及众多下游应用。为衡量效率，许多出版物依赖乘加运算次数（MACs）作为执行时间的预测指标。本文通过实验证明该指标存在缺陷，尤其在边缘设备场景下。通过对比常见架构设计元素的MACs数量与执行时间，我们识别出高效执行的关键因素，并为骨干网络设计优化提供见解。基于这些见解，我们提出LowFormer——一种新型视觉骨干网络家族。LowFormer采用精简的宏观与微观设计，包含Lowtention（多头自注意力机制的轻量化替代方案）。Lowtention不仅效率更高，还在ImageNet上取得更优结果。此外，我们推出了LowFormer的边缘GPU版本，可在边缘GPU和桌面GPU上进一步超越基线速度。通过在较小图像分类数据集上的评估，以及将其适配到目标检测、语义分割、图像检索、视觉目标跟踪等多个下游任务，我们验证了LowFormer的广泛适用性。与近期最先进骨干网络相比，LowFormer模型在各类硬件平台上均实现了显著加速。代码与模型详见链接。
URL
https://arxiv.org/abs/2603.26551
PDF
-
HINT: Composed Image Retrieval with Dual-path Compositional Contextualized Network
2026-03-27 12:01:40Mingyu Zhang, Zixu Li, Zhiwei Chen, Zhiheng Fu, Xiaowei Zhu, Jiajia Nie, Yinwei Wei, Yupeng HuAbstract
Composed Image Retrieval (CIR) is a challenging image retrieval paradigm. It aims to retrieve target images from large-scale image databases that are consistent with the modification semantics, based on a multimodal query composed of a reference image and modification text. Although existing methods have made significant progress in cross-modal alignment and feature fusion, a key flaw remains: the neglect of contextual information in discriminating matching samples. However, addressing this limitation is not an easy task due to two challenges: 1) implicit dependencies and 2) the lack of a differential amplification mechanism. To address these challenges, we propose a dual-patH composItional coNtextualized neTwork (HINT), which can perform contextualized encoding and amplify the similarity differences between matching and non-matching samples, thus improving the upper performance of CIR models in complex scenarios. Our HINT model achieves optimal performance on all metrics across two CIR benchmark datasets, demonstrating the superiority of our HINT model. Codes are available at this https URL.
Abstract (translated)
组合图像检索（CIR）是一种具有挑战性的图像检索范式，其目标是根据由参考图像和修改文本组成的多模态查询，从大规模图像数据库中检索与修改语义一致的目标图像。尽管现有方法在跨模态对齐与特征融合方面已取得显著进展，但仍存在一个关键缺陷：在判别匹配样本时忽略了上下文信息。然而，解决这一局限性并非易事，主要面临两大挑战：1）隐式依赖关系；2）缺乏差异放大机制。为应对这些挑战，我们提出了一种双路径组合上下文网络（HINT），该模型能够实现上下文化编码，并放大匹配样本与非匹配样本之间的相似度差异，从而提升CIR模型在复杂场景中的性能上限。我们的HINT模型在两个CIR基准数据集上的所有指标均达到最优性能，充分证明了其优越性。代码已公开于此链接。
URL
https://arxiv.org/abs/2603.26341
PDF
-
Few Shots Text to Image Retrieval: New Benchmarking Dataset and Optimization Methods
2026-03-26 20:26:05Ofer Idan, Vladi Vexler, Gil Lederman, Dima Sivov, Aviad Cohen Zada, Shir Niego KomfortiAbstract
Pre-trained vision-language models (VLMs) excel in multimodal tasks, commonly encoding images as embedding vectors for storage in databases and retrieval via approximate nearest neighbor search (ANNS). However, these models struggle with compositional queries and out-of-distribution (OOD) image-text pairs. Inspired by human cognition's ability to learn from minimal examples, we address this performance gap through few-shot learning approaches specifically designed for image retrieval. We introduce the Few-Shot Text-to-Image Retrieval (FSIR) task and its accompanying benchmark dataset, FSIR-BD - the first to explicitly target image retrieval by text accompanied by reference examples, focusing on the challenging compositional and OOD queries. The compositional part is divided to urban scenes and nature species, both in specific situations or with distinctive features. FSIR-BD contains 38,353 images and 303 queries, with 82% comprising the test corpus (averaging per query 37 positives, ground truth matches, and significant number of hard negatives) and 18% forming the few-shot reference corpus (FSR) of exemplar positive and hard negative images. Additionally, we propose two novel retrieval optimization methods leveraging single shot or few shot reference examples in the FSR to improve performance. Both methods are compatible with any pre-trained image encoder, making them applicable to existing large-scale environments. Our experiments demonstrate that: (1) FSIR-BD provides a challenging benchmark for image retrieval; and (2) our optimization methods outperform existing baselines as measured by mean Average Precision (mAP). Further research into FSIR optimization methods will help narrow the gap between machine and human-level understanding, particularly for compositional reasoning from limited examples.
Abstract (translated)
预训练视觉语言模型（VLMs）在多模态任务中表现出色，通常将图像编码为嵌入向量存储于数据库，并通过近似最近邻搜索（ANNS）进行检索。然而，这些模型在组合查询和分布外（OOD）图像-文本对方面存在不足。受人类通过少量样本学习能力的启发，我们通过专门针对图像检索的少样本学习方法来解决这一性能差距。我们提出了少样本文本到图像检索（FSIR）任务及其配套基准数据集FSIR-BD——这是首个明确针对附带参考示例的文本图像检索的数据集，重点关注具有挑战性的组合查询和分布外查询。组合部分细分为城市场景和自然物种，均涉及特定情境或显著特征。FSIR-BD包含38,353张图像和303个查询，其中82%构成测试语料库（每个查询平均包含37个正样本——即真实匹配，以及大量困难负样本），18%构成少样本参考语料库（FSR），包含示例正样本和困难负样本图像。此外，我们提出了两种新颖的检索优化方法，利用FSR中的单样本或少样本参考示例提升性能。这两种方法兼容任意预训练图像编码器，可应用于现有大规模环境。实验表明：（1）FSIR-BD为图像检索提供了具有挑战性的基准；（2）我们的优化方法在平均精度均值（mAP）指标上优于现有基线。对FSIR优化方法的进一步研究将有助于缩小机器与人类理解之间的差距，尤其在有限样本下的组合推理方面。
URL
https://arxiv.org/abs/2603.25891
PDF
-
TIGeR: A Unified Framework for Time, Images and Geo-location Retrieval
2026-03-25 19:20:48David G. Shatwell, Sirnam Swetha, Mubarak ShahAbstract
Many real-world applications in digital forensics, urban monitoring, and environmental analysis require jointly reasoning about visual appearance, geolocation, and time. Beyond standard geo-localization and time-of-capture prediction, these applications increasingly demand more complex capabilities, such as retrieving an image captured at the same location as a query image but at a specified target time. We formalize this problem as Geo-Time Aware Image Retrieval and curate a diverse benchmark of 4.5M paired image-location-time triplets for training and 86k high-quality triplets for evaluation. We then propose TIGeR, a multi-modal-transformer-based model that maps image, geolocation, and time into a unified geo-temporal embedding space. TIGeR supports flexible input configurations (single-modality and multi-modality queries) and uses the same representation to perform (i) geo-localization, (ii) time-of-capture prediction, and (iii) geo-time-aware retrieval. By better preserving underlying location identity under large appearance changes, TIGeR enables retrieval based on where and when a scene is, rather than purely on visual similarity. Extensive experiments show that TIGeR consistently outperforms strong baselines and state-of-the-art methods by up to 16% on time-of-year, 8% time-of-day prediction, and 14% in geo-time aware retrieval recall, highlighting the benefits of unified geo-temporal modeling.
Abstract (translated)
数字取证、城市监控和环境分析中的许多实际应用需要联合推理视觉外观、地理位置和时间信息。除标准地理定位和拍摄时间预测外，这些应用日益需要更复杂的能力，例如检索与查询图像拍摄于同一地点但在指定目标时间拍摄的图像。我们将此问题形式化为地理-时间感知图像检索，并构建了包含450万组图像-位置-时间三元组的多样化训练基准，以及8.6万组高质量三元组的评估基准。随后我们提出TIGeR，一种基于多模态Transformer的模型，可将图像、地理位置和时间映射到统一的地理-时间联合嵌入空间。TIGeR支持灵活的输入配置（单模态与多模态查询），并使用同一表征实现（i）地理定位、（ii）拍摄时间预测及（iii）地理-时间感知检索。通过在显著外观变化下更好地保持底层位置身份一致性，TIGeR实现了基于场景"何时何地"而非纯粹视觉相似度的检索。大量实验表明，TIGeR在季节时间预测上最高提升16%，昼夜时间预测提升8%，地理-时间感知检索召回率提升14%，持续优于强基线及现有先进方法，凸显了统一地理-时间建模的优势。
URL
https://arxiv.org/abs/2603.24749
PDF
-
Sparse Autoencoders for Interpretable Medical Image Representation Learning
2026-03-24 23:54:24Philipp Wesp, Robbie Holland, Vasiliki Sideri-Lampretsa, Sergios GatidisAbstract
Vision foundation models (FMs) achieve state-of-the-art performance in medical imaging. However, they encode information in abstract latent representations that clinicians cannot interrogate or verify. The goal of this study is to investigate Sparse Autoencoders (SAEs) for replacing opaque FM image representations with human-interpretable, sparse features. We train SAEs on embeddings from BiomedParse (biomedical) and DINOv3 (general-purpose) using 909,873 CT and MRI 2D image slices from the TotalSegmentator dataset. We find that learned sparse features: (a) reconstruct original embeddings with high fidelity (R2 up to 0.941) and recover up to 87.8% of downstream performance using only 10 features (99.4% dimensionality reduction), (b) preserve semantic fidelity in image retrieval tasks, (c) correspond to specific concepts that can be expressed in language using large language model (LLM)-based auto-interpretation. (d) bridge clinical language and abstract latent representations in zero-shot language-driven image retrieval. Our work indicates SAEs are a promising pathway towards interpretable, concept-driven medical vision systems. Code repository: this https URL.
Abstract (translated)
视觉基础模型（FMs）在医学影像领域达到了最先进的性能。然而，它们将信息编码在抽象的潜在表示中，临床医生无法对此进行 interrogate 或验证。本研究旨在探究稀疏自编码器（SAEs），用人类可解释的稀疏特征替代不透明的FM图像表示。我们在BiomedParse（生物医学）和DINOv3（通用）的嵌入表示上训练SAEs，使用TotalSegmentator数据集中的909,873张CT和MRI二维图像切片。我们发现学习到的稀疏特征：（a）以高保真度重建原始嵌入表示（R²最高达0.941），且仅使用10个特征（降低99.4%维度）即可恢复高达87.8%的下游任务性能；（b）在图像检索任务中保持语义保真度；（c）对应可通过基于大语言模型（LLM）的自动解释用语言表达的具体概念；（d）在零样本语言驱动图像检索中连接临床语言与抽象潜在表示。我们的工作表明SAEs是通向可解释的、概念驱动的医学视觉系统的一条有前景的路径。代码仓库：此https链接。
URL
https://arxiv.org/abs/2603.23794
PDF
-
ARGENT: Adaptive Hierarchical Image-Text Representations
2026-03-24 15:14:12Chuong Huynh, Hossein Souri, Abhinav Kumar, Vitali Petsiuk, Deen Dayal Mohan, Suren KumarAbstract
Large-scale Vision-Language Models (VLMs) such as CLIP learn powerful semantic representations but operate in Euclidean space, which fails to capture the inherent hierarchical structure of visual and linguistic concepts. Hyperbolic geometry, with its exponential volume growth, offers a principled alternative for embedding such hierarchies with low distortion. However, existing hyperbolic VLMs use entailment losses that are unstable: as parent embeddings contract toward the origin, their entailment cones widen toward a half-space, causing catastrophic cone collapse that destroys the intended hierarchy. Additionally, hierarchical evaluation of these models remains unreliable, being largely retrieval-based and correlation-based metrics and prone to taxonomy dependence and ambiguous negatives. To address these limitations, we propose an adaptive entailment loss paired with a norm regularizer that prevents cone collapse without heuristic aperture clipping. We further introduce an angle-based probabilistic entailment protocol (PEP) for evaluating hierarchical understanding, scored with AUC-ROC and Average Precision. This paper introduces a stronger hyperbolic VLM baseline ARGENT, Adaptive hieRarchical imaGe-tExt represeNTation. ARGENT improves the SOTA hyperbolic VLM by 0.7, 1.1, and 0.8 absolute points on image classification, text-to-image retrieval, and proposed hierarchical metrics, respectively.
Abstract (translated)
大规模视觉语言模型（如CLIP）能够学习强大的语义表征，但其在欧几里得空间中运作，无法捕捉视觉与语言概念固有的层次结构。双曲几何凭借其指数级增长的体积，为以低失真嵌入此类层次提供了原理性的替代方案。然而，现有双曲视觉语言模型所使用的蕴含损失存在不稳定性：当父级嵌入向原点收缩时，其蕴含锥会半空间方向扩张，导致灾难性的锥体塌陷，从而破坏预设的层次结构。此外，这些模型的层次化评估仍不可靠，主要依赖基于检索和相关性指标的方法，易受分类体系依赖与模糊负样本的影响。 为应对上述局限，本文提出一种自适应蕴含损失配合范数正则化器，可在无需启发式孔径裁剪的情况下防止锥体塌陷。同时引入基于角度的概率蕴含评估协议（PEP），采用AUC-ROC与平均精度进行评分。本研究提出了更优的双曲视觉语言模型基线ARGENT（自适应分层图像-文本表征）。ARGENT在图像分类、文本到图像检索以及提出的层次化指标上，分别较当前最优双曲视觉语言模型提升0.7、1.1和0.8个绝对点数。
URL
https://arxiv.org/abs/2603.23311
PDF
-
HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignment
2026-03-24 02:35:54Sangmin Jo, Wootaek Jeong, Da-Woon Heo, Yoohwan Hwang, Heung-Il SukAbstract
Recent progress in artificial intelligence has encouraged numerous attempts to understand and decode human visual system from brain signals. These prior works typically align neural activity independently with semantic and perceptual features extracted from images using pre-trained vision models. However, they fail to account for two key challenges: (1) the modality gap arising from the natural difference in the information level of representation between brain signals and images, and (2) the fact that semantic and perceptual features are highly entangled within neural activity. To address these issues, we utilize hyperbolic space, which is well-suited for considering differences in the amount of information and has the geometric property that geodesics between two points naturally bend toward the origin, where the representational capacity is lower. Leveraging these properties, we propose a novel framework, Hyperbolic Feature Interpolation (HyFI), which interpolates between semantic and perceptual visual features along hyperbolic geodesics. This enables both the fusion and compression of perceptual and semantic information, effectively reflecting the limited expressiveness of brain signals and the entangled nature of these features. As a result, it facilitates better alignment between brain and visual features. We demonstrate that HyFI achieves state-of-the-art performance in zero-shot brain-to-image retrieval, outperforming prior methods with Top-1 accuracy improvements of up to +17.3% on THINGS-EEG and +9.1% on THINGS-MEG.
Abstract (translated)
近年来人工智能的进展激励了众多通过脑信号理解和解码人类视觉系统的尝试。这些先前工作通常使用预训练的视觉模型从图像中提取语义和感知特征，并分别与神经活动进行对齐。然而，它们却未能解决两个关键挑战：（1）脑信号与图像在表征信息层级上存在天然差异而产生的模态差距；（2）语义和感知特征在神经活动中高度纠缠的事实。为应对这些问题，我们利用特别适合考虑信息量差异且具有几何特性的双曲空间——其两点间的测地线会自然向表征容量较低的原点弯曲。基于这些特性，我们提出了一种新颖框架——双曲特征插值（HyFI），该框架沿双曲测地线在语义与感知视觉特征间进行插值。这实现了感知与语义信息的融合与压缩，有效反映了脑信号有限的表达能力以及这些特征的纠缠本质，从而促进了脑特征与视觉特征间更优的对齐。我们证明HyFI在零样本脑到图像检索任务中达到了最先进性能，在THINGS-EEG数据集上Top-1准确率最高提升+17.3%，在THINGS-MEG数据集上最高提升+9.1%，超越了先前方法。
URL
https://arxiv.org/abs/2603.22721
PDF
-
ADaFuSE: Adaptive Diffusion-generated Image and Text Fusion for Interactive Text-to-Image Retrieval
2026-03-23 12:13:57Zhuocheng Zhang, Xingwu Zhang, Kangheng Liang, Guanxuan Li, Richard Mccreadie, Zijun LongAbstract
Recent advances in interactive text-to-image retrieval (I-TIR) use diffusion models to bridge the modality gap between the textual information need and the images to be searched, resulting in increased effectiveness. However, existing frameworks fuse multi-modal views of user feedback by simple embedding addition. In this work, we show that this static and undifferentiated fusion indiscriminately incorporates generative noise produced by the diffusion model, leading to performance degradation for up to 55.62% samples. We further propose ADaFuSE (Adaptive Diffusion-Text Fusion with Semantic-aware Experts), a lightweight fusion model designed to align and calibrate multi-modal views for diffusion-augmented I-TIR, which can be plugged into existing frameworks without modifying the backbone encoder. Specifically, we introduce a dual-branch fusion mechanism that employs an adaptive gating branch to dynamically balance modality reliability, alongside a semantic-aware mixture-of-experts branch to capture fine-grained cross-modal nuances. Via thorough evaluation over four standard I-TIR benchmarks, ADaFuSE achieves state-of-the-art performance, surpassing DAR by up to 3.49% in Hits@10 with only a 5.29% parameter increase, while exhibiting stronger robustness to noisy and longer interactive queries. These results show that generative augmentation coupled with principled fusion provides a simple, generalizable alternative to fine-tuning for interactive retrieval.
Abstract (translated)
近期交互式文本到图像检索（I-TIR）领域的进展利用扩散模型弥合文本信息需求与待检索图像之间的模态鸿沟，从而提升了检索效果。然而，现有框架通过简单的嵌入相加来融合用户反馈的多模态视图。本研究指出，这种静态且无差别的融合方式会不加分辨地纳入扩散模型产生的生成噪声，导致高达55.62%样本的性能下降。我们进一步提出ADaFuSE（基于语义感知专家的自适应扩散-文本融合），这是一个轻量级融合模型，旨在对齐和校准扩散增强型I-TIR的多模态视图，且无需修改主干编码器即可嵌入现有框架。具体而言，我们设计了双分支融合机制：自适应门控分支动态平衡模态可靠性，语义感知混合专家分支捕捉细粒度跨模态差异。在四个标准I-TIR基准上的全面评估表明，ADaFuSE达到最先进性能，在Hits@10指标上较DAR最高提升3.49%，而参数量仅增加5.29%，同时对噪声和长交互查询展现出更强鲁棒性。这些结果证明，生成增强结合原则性融合为交互式检索提供了一种无需微调的简单、可泛化方案。
URL
https://arxiv.org/abs/2603.21886
PDF
-
SATTC: Structure-Aware Label-Free Test-Time Calibration for Cross-Subject EEG-to-Image Retrieval
Abstract
Cross-subject EEG-to-image retrieval for visual decoding is challenged by subject shift and hubness in the embedding space, which distort similarity geometry and destabilize top-k rankings, making small-k shortlists unreliable. We introduce SATTC (Structure-Aware Test-Time Calibration), a label-free calibration head that operates directly on the similarity matrix of frozen EEG and image encoders. SATTC combines a geometric expert, subject-adaptive whitening of EEG embeddings with an adaptive variant of Cross-domain Similarity Local Scaling (CSLS), and a structural expert built from mutual nearest neighbors, bidirectional top-k ranks, and class popularity, fused via a simple Product-of-Experts rule. On THINGS-EEG under a strict leave-one-subject-out protocol, standardized inference with cosine similarities, L2-normalized embeddings, and candidate whitening already yields a strong cross-subject baseline over the original ATM retrieval setup. Building on this baseline, SATTC further improves Top-1 and Top-5 accuracy, reduces hubness and per-class imbalance, and produces more reliable small-k shortlists. These gains transfer across multiple EEG encoders, supporting SATTC as an encoder-agnostic, label-free test-time calibration layer for cross-subject neural decoding.
Abstract (translated)
跨被试脑电图-图像检索的视觉解码面临嵌入空间中的被试偏移与枢纽效应挑战，这些因素会扭曲相似性几何结构并 destabilize 前k名排序，导致小k候选列表不可靠。我们提出SATTC（结构感知测试时校准），这是一个无需标签的校准模块，可直接作用于冻结的脑电图与图像编码器的相似度矩阵。SATTC融合几何专家（对脑电图嵌入进行被试自适应白化处理，并结合跨域相似度局部缩放的 adaptive 变体）与结构专家（基于互最近邻、双向前k排序及类别流行度构建），通过简单的专家乘积规则进行融合。在THINGS-EEG数据集上采用严格的留一被试交叉验证协议时，使用余弦相似度标准化推理、L2归一化嵌入及候选白化处理，已能在原始ATM检索框架上建立强跨被试基线。在此基线基础上，SATTC进一步提升了Top-1与Top-5准确率，降低了枢纽效应与类别不平衡问题，并产生更可靠的小k候选列表。这些增益可迁移至多种脑电图编码器，支持SATTC作为跨被试神经解码中与编码器无关的免标签测试时校准层。
URL
https://arxiv.org/abs/2603.20738
PDF
-
A Multihead Continual Learning Framework for Fine-Grained Fashion Image Retrieval with Contrastive Learning and Exponential Moving Average Distillation
2026-03-21 04:58:02Ling Xiao, Toshihiko YamasakiAbstract
Most fine-grained fashion image retrieval (FIR) methods assume a static setting, requiring full retraining when new attributes appear, which is costly and impractical for dynamic scenarios. Although pretrained models support zero-shot inference, their accuracy drops without supervision, and no prior work explores class-incremental learning (CIL) for fine-grained FIR. We propose a multihead continual learning framework for fine-grained fashion image retrieval with contrastive learning and exponential moving average (EMA) distillation (MCL-FIR). MCL-FIR adopts a multi-head design to accommodate evolving classes across increments, reformulates triplet inputs into doublets with InfoNCE for simpler and more effective training, and employs EMA distillation for efficient knowledge transfer. Experiments across four datasets demonstrate that, beyond its scalability, MCL-FIR achieves a strong balance between efficiency and accuracy. It significantly outperforms CIL baselines under similar training cost, and compared with static methods, it delivers comparable performance while using only about 30% of the training cost. The source code is publicly available in this https URL.
Abstract (translated)
大多数细粒度时尚图像检索（FIR）方法假设静态设置，当新属性出现时需要完全重新训练，这在动态场景中成本高昂且不切实际。尽管预训练模型支持零样本推理，但在无监督情况下其准确率会下降，且尚无先前工作探索用于细粒度FIR的类增量学习（CIL）。我们提出了一种结合对比学习与指数移动平均（EMA）蒸馏的多头部持续学习框架（MCL-FIR），用于细粒度时尚图像检索。MCL-FIR采用多头部设计以适应增量过程中不断演变的类别，将三元组输入重构为双元组并采用InfoNCE损失以实现更简洁有效的训练，同时利用EMA蒸馏实现高效的知识迁移。在四个数据集上的实验表明，除可扩展性外，MCL-FIR在效率与准确率间取得了强平衡：在相似训练成本下显著优于CIL基线方法；与静态方法相比，其性能相当但训练成本仅约30%。源代码已公开于此链接。
URL
https://arxiv.org/abs/2603.20648
PDF
-
IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment
2026-03-20 11:25:55Simone Magistri, Dipam Goswami, Marco Mistretta, Bart{\l}omiej Twardowski, Joost van de Weijer, Andrew D. BagdanovAbstract
Vision-Language Models like CLIP are extensively used for inter-modal tasks which involve both visual and text modalities. However, when the individual modality encoders are applied to inherently intra-modal tasks like image-to-image retrieval, their performance suffers from the intra-modal misalignment. In this paper we study intra-modal misalignment in CLIP with a focus on the role of the projectors that map pre-projection image and text embeddings into the shared embedding space. By analyzing the form of the cosine similarity applied to projected features, and its interaction with the contrastive CLIP loss, we show that there is an inter-modal operator responsible for aligning the two modalities during training, and a second, intra-modal operator that only enforces intra-modal normalization but does nothing to promote intra-modal alignment. Via spectral analysis of the inter-modal operator, we identify an approximately isotropic subspace in which the two modalities are well-aligned, as well as anisotropic directions specific to each modality. We demonstrate that this aligned subspace can be directly obtained from the projector weights and that removing the anisotropic directions improves intra-modal alignment. Our experiments on intra-modal retrieval and classification benchmarks show that our training-free method reduces intra-modal misalignment, greatly lowers latency, and outperforms existing approaches across multiple pre-trained CLIP-like models. The code is publicly available at: this https URL.
Abstract (translated)
像CLIP这样的视觉-语言模型被广泛用于涉及视觉和文本模态的跨模态任务。然而，当单独的模态编码器应用于图像到图像检索等固有的模态内任务时，其性能会受到模态内错位的影响。本文研究了CLIP中的模态内错位问题，重点关注将预投影图像和文本嵌入映射到共享嵌入空间的投影器的作用。通过分析应用于投影特征的余弦相似度的形式及其与对比式CLIP损失的相互作用，我们发现在训练过程中存在一个负责对齐两个模态的跨模态算子，以及第二个仅强制执行模态内归一化但无助于促进模态内对齐的模态内算子。通过对跨模态算子进行谱分析，我们识别出一个近似各向同性的子空间，其中两个模态对齐良好，以及每个模态特有的各向异性方向。我们证明这个对齐子空间可以直接从投影器权重中获得，并且去除各向异性方向可以改善模态内对齐。我们在模态内检索和分类基准上的实验表明，我们这种无需训练的方法减少了模态内错位，显著降低了延迟，并在多个预训练的类CLIP模型上优于现有方法。代码已在以下网址公开提供：此https链接。
URL
https://arxiv.org/abs/2603.19862
PDF
-
MCoT-MVS: Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning for Composed Image Retrieval
2026-03-18 04:49:19Xuri Ge, Chunhao Wang, Xindi Wang, Zheyun Qin, Zhumin Chen, Xin XinAbstract
Composed Image Retrieval (CIR) aims to retrieve target images based on a reference image and modified texts. However, existing methods often struggle to extract the correct semantic cues from the reference image that best reflect the user's intent under textual modification prompts, resulting in interference from irrelevant visual noise. In this paper, we propose a novel Multi-level Vision Selection by Multi-modal Chain-of-Thought Reasoning (MCoT-MVS) for CIR, integrating attention-aware multi-level vision features guided by reasoning cues from a multi-modal large language model (MLLM). Specifically, we leverage an MLLM to perform chain-of-thought reasoning on the multimodal composed input, generating the retained, removed, and target-inferred texts. These textual cues subsequently guide two reference visual attention selection modules to selectively extract discriminative patch-level and instance-level semantics from the reference image. Finally, to effectively fuse these multi-granular visual cues with the modified text and the imagined target description, we design a weighted hierarchical combination module to align the composed query with target images in a unified embedding space. Extensive experiments on two CIR benchmarks, namely CIRR and FashionIQ, demonstrate that our approach consistently outperforms existing methods and achieves new state-of-the-art performance. Code and trained models are publicly released.
Abstract (translated)
URL
https://arxiv.org/abs/2603.17360
PDF
