Abstract
Multi-view person association is a fundamental step towards multi-view analysis of human activities. Although the person re-identification features have been proven effective, they become unreliable in challenging scenes where persons share similar appearances. Therefore, cross-view geometric constraints are required for a more robust association. However, most existing approaches are either fully-supervised using ground-truth identity labels or require calibrated camera parameters that are hard to obtain. In this work, we investigate the potential of learning from synchronization, and propose a self-supervised uncalibrated multi-view person association approach, Self-MVA, without using any annotations. Specifically, we propose a self-supervised learning framework, consisting of an encoder-decoder model and a self-supervised pretext task, cross-view image synchronization, which aims to distinguish whether two images from different views are captured at the same time. The model encodes each person's unified geometric and appearance features, and we train it by utilizing synchronization labels for supervision after applying Hungarian matching to bridge the gap between instance-wise and image-wise distances. To further reduce the solution space, we propose two types of self-supervised linear constraints: multi-view re-projection and pairwise edge association. Extensive experiments on three challenging public benchmark datasets (WILDTRACK, MVOR, and SOLDIERS) show that our approach achieves state-of-the-art results, surpassing existing unsupervised and fully-supervised approaches. Code is available at this https URL.
Abstract (translated)
多视角人物关联是人类活动多视角分析中的基础步骤。虽然人员再识别特征已被证明有效，但在场景中人物外观相似时，这些特征变得不可靠。因此，跨视图几何约束对于实现更稳健的关联是必要的。然而，大多数现有方法要么完全依赖于使用真实身份标签进行监督训练，要么需要难以获取的校准摄像机参数。在本文工作中，我们研究了从同步学习中的潜力，并提出了一种无需任何注释的自监督未校准多视角人物关联方法——Self-MVA。 具体来说，我们提出了一个包含编码器-解码器模型和自我监督前置任务（跨视图图像同步）的自监督学习框架。此前置任务旨在区分来自不同视角的两张图片是否在同一时间被捕捉。该模型编码每个人统一的几何与外观特征，并通过应用匈牙利匹配算法，利用同步标签进行训练以弥合实例级和图像级距离之间的差距。 为了进一步缩小解决方案空间，我们提出了两种自监督线性约束类型：多视图重新投影和成对边缘关联。 在三个具有挑战性的公开基准数据集（WILDTRACK、MVOR 和 SOLDIERS）上的大量实验表明，我们的方法取得了最先进的结果，超过了现有的无监督和完全监督的方法。代码可在提供的链接中获取。
URL
https://arxiv.org/abs/2503.13739
