Abstract
The Visual Language Model, known for its robust cross-modal capabilities, has been extensively applied in various computer vision tasks. In this paper, we explore the use of CLIP (Contrastive Language-Image Pretraining), a vision-language model pretrained on large-scale image-text pairs to align visual and textual features, for acquiring fine-grained and domain-invariant representations in generalizable person re-identification. The adaptation of CLIP to the task presents two primary challenges: learning more fine-grained features to enhance discriminative ability, and learning more domain-invariant features to improve the model's generalization capabilities. To mitigate the first challenge thereby enhance the ability to learn fine-grained features, a three-stage strategy is proposed to boost the accuracy of text descriptions. Initially, the image encoder is trained to effectively adapt to person re-identification tasks. In the second stage, the features extracted by the image encoder are used to generate textual descriptions (i.e., prompts) for each image. Finally, the text encoder with the learned prompts is employed to guide the training of the final image encoder. To enhance the model's generalization capabilities to unseen domains, a bidirectional guiding method is introduced to learn domain-invariant image features. Specifically, domain-invariant and domain-relevant prompts are generated, and both positive (pulling together image features and domain-invariant prompts) and negative (pushing apart image features and domain-relevant prompts) views are used to train the image encoder. Collectively, these strategies contribute to the development of an innovative CLIP-based framework for learning fine-grained generalized features in person re-identification.
Abstract (translated)
视觉语言模型因其强大的跨模态能力而著称，在各种计算机视觉任务中得到了广泛应用。本文探讨了将CLIP（对比语言图像预训练）应用于一般化的人体再识别任务，以获取细粒度和领域不变的表示形式。CLIP是一个基于大规模图像-文本对进行预训练的视觉-语言模型，旨在对齐视觉和文本特征。 在该任务中应用CLIP面临两大主要挑战：一是学习更细粒度的特征以增强区分能力；二是学习更多领域不变的特征以提升模型的一般化能力。为缓解第一个挑战并提高学习细粒度特征的能力，本文提出了一种三阶段策略来提升文本描述的准确性。首先训练图像编码器有效适应人体再识别任务。在第二阶段中，通过图像编码器提取的特征生成每张图片的文字描述（即提示）。最后，使用带有已学得提示的文本编码器来指导最终图像编码器的训练。 为了增强模型对未见领域的泛化能力，引入了一种双向引导方法来学习领域不变的图像特征。具体而言，该方法通过正向视角（拉近图像特征和领域不变提示）以及反向视角（推远图像特征和领域相关的提示）生成领域不变与相关提示，并以此训练图像编码器。 综合上述策略，本文发展了一个基于CLIP的创新框架，在人体再识别任务中学习细粒度且具有泛化能力的特征。
URL
https://arxiv.org/abs/2501.16065
