Abstract
Vision Foundation Model (VFM) such as the Segment Anything Model (SAM) and Contrastive Language-Image Pre-training Model (CLIP) has shown promising performance for segmentation and detection tasks. However, although SAM excels in fine-grained segmentation, it faces major challenges when applying it to semantic-aware segmentation. While CLIP exhibits a strong semantic understanding capability via aligning the global features of language and vision, it has deficiencies in fine-grained segmentation tasks. Human parsing requires to segment human bodies into constituent parts and involves both accurate fine-grained segmentation and high semantic understanding of each part. Based on traits of SAM and CLIP, we formulate high efficient modules to effectively integrate features of them to benefit human parsing. We propose a Semantic-Refinement Module to integrate semantic features of CLIP with SAM features to benefit parsing. Moreover, we formulate a high efficient Fine-tuning Module to adjust the pretrained SAM for human parsing that needs high semantic information and simultaneously demands spatial details, which significantly reduces the training time compared with full-time training and achieves notable performance. Extensive experiments demonstrate the effectiveness of our method on LIP, PPP, and CIHP databases.
Abstract (translated)
翻译如下： 基于视觉基础模型（VFM）如分割任何模型（SAM）和对比语言-图像预训练模型（CLIP）在分割和检测任务中展现出了令人鼓舞的性能。然而，虽然SAM在细粒度分割方面表现出色，但在应用于语义感知分割时却面临重大挑战。同时，尽管CLIP通过将语言和视觉的全局特征对齐来展示出强大的语义理解能力，但它在细粒度分割任务中的表现不佳。人体解析需要对人体进行成分划分，并且既要求准确的细粒度分割又需要各部分具有高度的语义理解。基于SAM和CLIP的特点，我们设计了高效的模块以有效地整合它们的功能，从而有益于人体解析。我们提出了一种语义精炼模块（Semantic-Refinement Module），将CLIP的语义特征与SAM功能相结合，有利于解析。此外，我们还制定了一个高效微调模块，用于调整预训练的SAM，使其适应需要高度语义信息和同时要求空间细节的人体解析任务，相较于完全重新训练显著减少了训练时间，并且取得了显著性能。广泛的实验在LIP、PPP和CIHP数据库上证明了该方法的有效性。
URL
https://arxiv.org/abs/2503.22237
