Abstract
Recent advancements in adapting vision-language pre-training models like CLIP for person re-identification (ReID) tasks often rely on complex adapter design or modality-specific tuning while neglecting cross-modal interaction, leading to high computational costs or suboptimal alignment. To address these limitations, we propose a simple yet effective framework named Selective Cross-modal Prompt Tuning (SCING) that enhances cross-modal alignment and robustness against real-world perturbations. Our method introduces two key innovations: Firstly, we proposed Selective Visual Prompt Fusion (SVIP), a lightweight module that dynamically injects discriminative visual features into text prompts via a cross-modal gating mechanism. Moreover, the proposed Perturbation-Driven Consistency Alignment (PDCA) is a dual-path training strategy that enforces invariant feature alignment under random image perturbations by regularizing consistency between original and augmented cross-modal embeddings. Extensive experiments are conducted on several popular benchmarks covering Market1501, DukeMTMC-ReID, Occluded-Duke, Occluded-REID, and P-DukeMTMC, which demonstrate the impressive performance of the proposed method. Notably, our framework eliminates heavy adapters while maintaining efficient inference, achieving an optimal trade-off between performance and computational overhead. The code will be released upon acceptance.
Abstract (translated)
近期，针对利用如CLIP等视觉-语言预训练模型进行人员重新识别（ReID）任务的研究进展，通常依赖于复杂的适配器设计或特定模态的调优方法，但这些方法往往忽视了跨模态交互的重要性，导致计算成本高昂或者对齐效果不佳。为了解决这些问题，我们提出了一种简单而有效的框架，名为选择性跨模态提示调整（SCING），该框架旨在增强跨模态对齐能力并提升在现实世界扰动下的鲁棒性。 我们的方法引入了两个关键创新点： 1. **选择性视觉提示融合（SVIP）**：这是一种轻量级模块，通过一个跨模态门控机制动态地将具有区分性的视觉特征注入文本提示中。 2. **扰动驱动的一致性对齐（PDCA）**：这是一种双路径训练策略，通过对原始和增强后的跨模态嵌入之间的一致性进行正则化来强制执行在随机图像扰动下的不变特性对齐。 我们在多个流行基准数据集上进行了广泛的实验，包括Market1501、DukeMTMC-ReID、Occluded-Duke、Occluded-REID和P-DukeMTMC。实验结果表明了所提出方法的卓越性能。值得注意的是，我们的框架在消除重适配器的同时保持高效的推理能力，从而实现了性能与计算开销之间的最佳平衡。 该代码将在论文被接受后发布。
URL
https://arxiv.org/abs/2507.00506
