Abstract
Reasoning over visual relationships-spatial, functional, interactional, social, etc.-is considered to be a fundamental component of human cognition. Yet, despite the major advances in visual comprehension in multimodal language models (MLMs), precise reasoning over relationships and their generations remains a challenge. We introduce ROBIN: an MLM instruction-tuned with densely annotated relationships capable of constructing high-quality dense scene graphs at scale. To train ROBIN, we curate SVG, a synthetic scene graph dataset by completing the missing relations of selected objects in existing scene graphs using a teacher MLM and a carefully designed filtering process to ensure high-quality. To generate more accurate and rich scene graphs at scale for any image, we introduce SG-EDIT: a self-distillation framework where GPT-4o further refines ROBIN's predicted scene graphs by removing unlikely relations and/or suggesting relevant ones. In total, our dataset contains 146K images and 5.6M relationships for 2.6M objects. Results show that our ROBIN-3B model, despite being trained on less than 3 million instances, outperforms similar-size models trained on over 300 million instances on relationship understanding benchmarks, and even surpasses larger models up to 13B parameters. Notably, it achieves state-of-the-art performance in referring expression comprehension with a score of 88.9, surpassing the previous best of 87.4. Our results suggest that training on the refined scene graph data is crucial to maintaining high performance across diverse visual reasoning task.
Abstract (translated)
基于视觉关系（如空间、功能、交互和社会关系）的推理被认为是人类认知的基本组成部分。尽管多模态语言模型（MLMs）在视觉理解方面取得了重大进展，但在准确推断关系及其生成上仍面临挑战。为此，我们引入了ROBIN：这是一种通过密集注释的关系进行微调的MLM，能够大规模构建高质量的密集场景图。 为了训练ROBIN，我们整理了SVG数据集——这是一个合成的场景图数据集。该数据集通过对现有场景图中选定对象缺失关系的补全而生成，使用教师MLM和精心设计的过滤过程以确保高质量的数据质量。为在任何图像上生成更准确、更丰富的场景图，我们引入了一个自我蒸馏框架SG-EDIT：通过GPT-4o进一步精炼ROBIN预测的场景图，删除不太可能的关系并/或建议相关关系。 总计，我们的数据集包含146K张图片和5.6M种关系，涉及2.6M个对象。实验结果表明，尽管训练样本少于300万，但我们的ROBIN-3B模型在关系理解基准测试中超越了相同大小的、使用超过3亿实例进行训练的模型，并且甚至超过了多达130亿参数的大规模模型。特别值得一提的是，在指代表达理解任务上取得了88.9的成绩（前最佳成绩为87.4），达到了当前最优水平。我们的研究结果表明，利用经过精炼的场景图数据对模型进行训练对于在各种视觉推理任务中保持高水平性能至关重要。
URL
https://arxiv.org/abs/2506.07643
