Abstract
Action-conditioned video prediction models (often referred to as world models) have shown strong potential for robotics applications, but existing approaches are often slow and struggle to capture physically consistent interactions over long horizons, limiting their usefulness for scalable robot policy training and evaluation. We present Interactive World Simulator, a framework for building interactive world models from a moderate-sized robot interaction dataset. Our approach leverages consistency models for both image decoding and latent-space dynamics prediction, enabling fast and stable simulation of physical interactions. In our experiments, the learned world models produce interaction-consistent pixel-level predictions and support stable long-horizon interactions for more than 10 minutes at 15 FPS on a single RTX 4090 GPU. Our framework enables scalable demonstration collection solely within the world models to train state-of-the-art imitation policies. Through extensive real-world evaluation across diverse tasks involving rigid objects, deformable objects, object piles, and their interactions, we find that policies trained on world-model-generated data perform comparably to those trained on the same amount of real-world data. Additionally, we evaluate policies both within the world models and in the real world across diverse tasks, and observe a strong correlation between simulated and real-world performance. Together, these results establish the Interactive World Simulator as a stable and physically consistent surrogate for scalable robotic data generation and faithful, reproducible policy evaluation.
Abstract (translated)
基于动作条件的视频预测模型（通常称为世界模型）在机器人应用中显示出强大的潜力，但现有的方法往往速度较慢，并且难以捕捉长时间范围内的物理一致交互，这限制了它们对可扩展机器人策略训练和评估的实用性。我们提出了互动世界模拟器（Interactive World Simulator），这是一个用于从适度大小的机器人互动数据集中构建互动世界模型的框架。我们的方法利用一致性模型进行图像解码以及隐空间动态预测，从而能够快速且稳定地模拟物理交互。在实验中，所学习的世界模型能够产生交互一致的像素级预测，并支持在一个RTX 4090 GPU上以每秒15帧的速度进行超过十分钟的长时间范围内的稳定互动。 我们的框架允许完全在世界模型内部收集大规模演示数据来训练最先进的模仿策略。通过跨涉及刚性物体、可变形物体、堆积物及其相互作用等多种任务的真实世界评估，我们发现基于世界模型生成的数据训练出的策略与同样数量真实世界数据训练出来的策略表现相当。此外，我们在世界模型和现实世界中对各种任务进行策略评价，并观察到模拟性能与实际世界性能之间存在强烈的关联性。 综上所述，这些结果确立了互动世界模拟器作为一个稳定且物理一致的替代方案，在大规模机器人数据生成以及忠实、可重复的策略评估方面具有重要作用。
URL
https://arxiv.org/abs/2603.08546
