Abstract
Video-based visible-infrared person re-identification (VVI-ReID) is challenging due to significant modality feature discrepancies. Spatial-temporal information in videos is crucial, but the accuracy of spatial-temporal information is often influenced by issues like low quality and occlusions in videos. Existing methods mainly focus on reducing modality differences, but pay limited attention to improving spatial-temporal features, particularly for infrared videos. To address this, we propose a novel Skeleton-guided spatial-Temporal feAture leaRning (STAR) method for VVI-ReID. By using skeleton information, which is robust to issues such as poor image quality and occlusions, STAR improves the accuracy of spatial-temporal features in videos of both modalities. Specifically, STAR employs two levels of skeleton-guided strategies: frame level and sequence level. At the frame level, the robust structured skeleton information is used to refine the visual features of individual frames. At the sequence level, we design a feature aggregation mechanism based on skeleton key points graph, which learns the contribution of different body parts to spatial-temporal features, further enhancing the accuracy of global features. Experiments on benchmark datasets demonstrate that STAR outperforms state-of-the-art methods. Code will be open source soon.
Abstract (translated)
基于视频的可见光-红外行人重识别（VVI-ReID）因其显著的模态特征差异而具有挑战性。视频中的时空信息至关重要，但其准确性常受视频质量低和遮挡等问题的影响。现有方法主要集中在减少模态差异上，但对于改善时空特性，尤其是对于红外视频方面的关注较少。为了解决这一问题，我们提出了一种用于VVI-ReID的新型骨骼引导的空间-时间特征学习（STAR）方法。通过使用对图像质量问题和遮挡具有鲁棒性的骨架信息，STAR提高了两种模态视频中空间-时间特性的准确性。具体而言，STAR采用了两个层次的骨骼引导策略：帧级和序列级。在帧级上，利用鲁棒的结构化骨架信息来优化单个帧的视觉特征；在序列级上，我们设计了一种基于骨架关键点图的特征聚合机制，该机制学习不同身体部位对空间-时间特性贡献的学习，进一步提高了全局特性的准确性。基准数据集上的实验表明，STAR优于现有最先进的方法。代码即将开源。
URL
https://arxiv.org/abs/2411.11069
