Abstract
Video-based Visible-Infrared Person Re-Identification (VVI-ReID) aims to match pedestrian sequences across modalities by extracting modality-invariant sequence-level features. As a high-level semantic representation, language provides a consistent description of pedestrian characteristics in both infrared and visible modalities. Leveraging the Contrastive Language-Image Pre-training (CLIP) model to generate video-level language prompts and guide the learning of modality-invariant sequence-level features is theoretically feasible. However, the challenge of generating and utilizing modality-shared video-level language prompts to address modality gaps remains a critical problem. To address this problem, we propose a simple yet powerful framework, video-level language-driven VVI-ReID (VLD), which consists of two core modules: invariant-modality language prompting (IMLP) and spatial-temporal prompting (STP). IMLP employs a joint fine-tuning strategy for the visual encoder and the prompt learner to effectively generate modality-shared text prompts and align them with visual features from different modalities in CLIP's multimodal space, thereby mitigating modality differences. Additionally, STP models spatiotemporal information through two submodules, the spatial-temporal hub (STH) and spatial-temporal aggregation (STA), which further enhance IMLP by incorporating spatiotemporal information into text prompts. The STH aggregates and diffuses spatiotemporal information into the [CLS] token of each frame across the vision transformer (ViT) layers, whereas STA introduces dedicated identity-level loss and specialized multihead attention to ensure that the STH focuses on identity-relevant spatiotemporal feature aggregation. The VLD framework achieves state-of-the-art results on two VVI-ReID benchmarks. The code will be released at this https URL.
Abstract (translated)
基于视频的可见光-红外行人再识别（VVI-ReID）旨在通过提取模态不变的序列级特征来匹配不同模态下的行人序列。作为一种高级语义表示，语言为可见光和红外两种模态下的人行特性提供了一致性的描述。理论上，利用对比学习的语言图像预训练（CLIP）模型生成视频级别的文本提示，并指导模态不变序列级别特征的学习是可行的。然而，如何生成并使用跨模态共享的视频级语言提示以解决不同模态之间的差距问题仍是一个关键挑战。 为了解决这一问题，我们提出了一种简单而强大的框架——视频级别语言驱动的VVI-ReID（VLD），该框架由两个核心模块组成：不变性模态语言提示（IMLP）和时空提示（STP）。IMLP采用视觉编码器和提示学习者的联合微调策略，有效生成跨模态共享的文字提示，并在CLIP的多模态空间中使其与不同模态下的视觉特征对齐，从而缓解了模态差异。此外，通过空间-时间枢纽（STH）和时空聚合（STA）这两个子模块，STP建模空间-时间信息，并进一步通过将空间-时间信息融入文本提示来增强IMLP。 具体来说，STH在每个帧的[CLS]标记上跨视觉变换器（ViT）层进行空间-时间信息的聚集和扩散操作；而STA则引入了专门的身份级别损失以及专业化的多头注意力机制，以确保STH专注于身份相关的时间空间特征聚合。VLD框架在两个VVI-ReID基准测试中取得了最先进的结果。 有关此项目的代码将在此 URL 发布：[请参阅原文链接]
URL
https://arxiv.org/abs/2506.02439
