Abstract
Existing methods for human parsing into body parts and clothing often use fixed mask categories with broad labels that obscure fine-grained clothing types. Recent open-vocabulary segmentation approaches leverage pretrained text-to-image (T2I) diffusion model features for strong zero-shot transfer, but typically group entire humans into a single person category, failing to distinguish diverse clothing or detailed body parts. To address this, we propose Spectrum, a unified network for part-level pixel parsing (body parts and clothing) and instance-level grouping. While diffusion-based open-vocabulary models generalize well across tasks, their internal representations are not specialized for detailed human parsing. We observe that, unlike diffusion models with broad representations, image-driven 3D texture generators maintain faithful correspondence to input images, enabling stronger representations for parsing diverse clothing and body parts. Spectrum introduces a novel repurposing of an Image-to-Texture (I2Tx) diffusion model -- obtained by fine-tuning a T2I model on 3D human texture maps -- for improved alignment with body parts and clothing. From an input image, we extract human-part internal features via the I2Tx diffusion model and generate semantically valid masks aligned to diverse clothing categories through prompt-guided grounding. Once trained, Spectrum produces semantic segmentation maps for every visible body part and clothing category, ignoring standalone garments or irrelevant objects, for any number of humans in the scene. We conduct extensive cross-dataset experiments -- separately assessing body parts, clothing parts, unseen clothing categories, and full-body masks -- and demonstrate that Spectrum consistently outperforms baseline methods in prompt-based segmentation.
Abstract (translated)
现有的人体解析方法通常使用固定的具有广泛标签的掩码类别，这些标签遮蔽了精细的服装类型。最近的开放词汇分割方法利用预训练的文本到图像（T2I）扩散模型特征实现了强大的零样本迁移能力，但它们通常将整个个体归类为单一的人体类别，无法区分多样化的服装或详细的身体部位。为了应对这一挑战，我们提出了Spectrum，这是一个统一的网络，用于部分级别的像素解析（包括身体部位和服装）以及实例级别的分组。 尽管基于扩散模型的开放词汇方法在跨任务中表现出色，但它们内部表示并不专门针对详细的人体解析而优化。我们观察到，与具有广泛表征的扩散模型不同，图像驱动的3D纹理生成器与其输入图像保持忠实对应关系，这使得它们能够为多样化的服装和身体部位提供更强的表征能力。Spectrum引入了一个新颖的应用：通过在3D人体纹理图上微调T2I模型获得的图像到纹理（I2Tx）扩散模型，以实现更好的与身体部位和服装的对齐。 从输入图像中，我们通过I2Tx扩散模型提取人体部分内部特征，并生成语义有效且与多种服装类别相对应的掩码，这由提示引导定位完成。训练完成后，Spectrum能够为场景中的每个可见身体部分和服装类别产生语义分割图，忽略单独的衣物或不相关的物体，无论场景中的人数多少。 我们进行了广泛的跨数据集实验——分别评估身体部位、服装部件、未见的服装类别以及全身掩码，并证明了在基于提示的分割任务上Spectrum始终优于基线方法。
URL
https://arxiv.org/abs/2508.06032
