Abstract
Multi-modal large language models (MLLMs) are making rapid progress toward general-purpose embodied agents. However, current training pipelines primarily rely on high-level vision-sound-text pairs and lack fine-grained, structured alignment between pixel-level visual content and textual semantics. To overcome this challenge, we propose ESCA, a new framework for contextualizing embodied agents through structured spatial-temporal understanding. At its core is SGClip, a novel CLIP-based, open-domain, and promptable model for generating scene graphs. SGClip is trained on 87K+ open-domain videos via a neurosymbolic learning pipeline, which harnesses model-driven self-supervision from video-caption pairs and structured reasoning, thereby eliminating the need for human-labeled scene graph annotations. We demonstrate that SGClip supports both prompt-based inference and task-specific fine-tuning, excelling in scene graph generation and action localization benchmarks. ESCA with SGClip consistently improves both open-source and commercial MLLMs, achieving state-of-the-art performance across two embodied environments. Notably, it significantly reduces agent perception errors and enables open-source models to surpass proprietary baselines.
Abstract (translated)
多模态大型语言模型（MLLMs）正在迅速向通用实体代理发展。然而，当前的训练流程主要依赖于高层次的视觉-声音-文本对，缺乏像素级视觉内容与文本语义之间的细粒度、结构化对齐。为解决这一挑战，我们提出了ESCA，这是一个通过结构化的时空理解来赋予实体代理人情境的新框架。其核心是SGClip，这是一种基于CLIP（对比语言-图像预训练模型）、开放领域且可提示的模型，用于生成场景图。SGClip在87K+个开放领域的视频上进行训练，通过神经符号学习流程，利用从视频字幕对中得出的模型驱动自我监督和结构化推理，从而无需人类标注的场景图注释。我们证明了SGClip支持基于提示的推理和特定任务的微调，在场景图生成和动作定位基准测试中表现出色。带有SGClip的ESCA在两个实体环境中持续提升了开源及商业MLLMs的表现，达到了最先进的性能水平，并显著减少了代理人的感知错误，使开源模型能够超越专有基线。
URL
https://arxiv.org/abs/2510.15963
