Abstract
Scene Graph Generation (SGG) encodes visual relationships between objects in images as graph structures. Thanks to the advances of Vision-Language Models (VLMs), the task of Open-Vocabulary SGG has been recently proposed where models are evaluated on their functionality to learn a wide and diverse range of relations. Current benchmarks in SGG, however, possess a very limited vocabulary, making the evaluation of open-source models inefficient. In this paper, we propose a new reference-free metric to fairly evaluate the open-vocabulary capabilities of VLMs for relation prediction. Another limitation of Open-Vocabulary SGG is the reliance on weakly supervised data of poor quality for pre-training. We also propose a new solution for quickly generating high-quality synthetic data through region-specific prompt tuning of VLMs. Experimental results show that pre-training with this new data split can benefit the generalization capabilities of Open-Voc SGG models.
Abstract (translated)
场景图生成（SGG）任务旨在将图像中物体之间的视觉关系编码为图形结构。得益于视觉-语言模型（VLMs）的进步，最近提出了开放词汇表的SGG任务，评估模型学习广泛且多样化的关系的能力。然而，现有的SGG基准测试具有的词汇量非常有限，这使得开源模型的效果评估变得效率低下。 本文中，我们提出了一种新的无参考度量标准，旨在公平地评估VLM在关系预测中的开放词汇表能力。另外，开放词汇表的SGG任务还受限于依赖质量较差的弱监督数据进行预训练的问题。为此，我们也提出了一个新方案，通过区域特定提示调优（region-specific prompt tuning）的方法快速生成高质量的合成数据。 实验结果表明，使用这种新的数据划分方式进行预训练可以提升开放词汇表SGG模型的泛化能力。
URL
https://arxiv.org/abs/2509.01209
