Abstract
Multi-modal object Re-Identification (ReID) aims to exploit complementary information from different modalities to retrieve specific objects. However, existing methods often rely on hard token filtering or simple fusion strategies, which can lead to the loss of discriminative cues and increased background interference. To address these challenges, we propose STMI, a novel multi-modal learning framework consisting of three key components: (1) Segmentation-Guided Feature Modulation (SFM) module leverages SAM-generated masks to enhance foreground representations and suppress background noise through learnable attention modulation; (2) Semantic Token Reallocation (STR) module employs learnable query tokens and an adaptive reallocation mechanism to extract compact and informative representations without discarding any tokens; (3) Cross-Modal Hypergraph Interaction (CHI) module constructs a unified hypergraph across modalities to capture high-order semantic relationships. Extensive experiments on public benchmarks (i.e., RGBNT201, RGBNT100, and MSVR310) demonstrate the effectiveness and robustness of our proposed STMI framework in multi-modal ReID scenarios.
Abstract (translated)
多模态对象重识别（ReID）的目标是利用不同模态提供的互补信息来检索特定的对象。然而，现有的方法往往依赖于硬令牌过滤或简单的融合策略，这可能导致判别线索的丢失和背景干扰的增加。为了解决这些挑战，我们提出了STMI，这是一种新颖的多模态学习框架，包含三个关键组成部分： 1. **分割引导特征调制（SFM）模块**：该模块利用SAM生成的掩码来增强前景表示并抑制背景噪声，通过可学习的注意力调节实现。 2. **语义令牌重分配（STR）模块**：此模块采用可学习查询令牌和自适应重分配机制，在不丢弃任何令牌的情况下提取紧凑且具有信息量的表示形式。 3. **跨模态超图交互（CHI）模块**：该模块构建了一个跨模态统一超图，以捕捉高阶语义关系。 在公共基准测试（例如RGBNT201、RGBNT100和MSVR310）上的广泛实验表明，我们提出的STMI框架在多模态ReID场景中具有有效性和鲁棒性。
URL
https://arxiv.org/abs/2603.00695
