Abstract
We present DINO-world, a powerful generalist video world model trained to predict future frames in the latent space of DINOv2. By leveraging a pre-trained image encoder and training a future predictor on a large-scale uncurated video dataset, DINO-world learns the temporal dynamics of diverse scenes, from driving and indoor scenes to simulated environments. We show that DINO-world outperforms previous models on a variety of video prediction benchmarks, e.g. segmentation and depth forecasting, and demonstrates strong understanding of intuitive physics. Furthermore, we show that it is possible to fine-tune the predictor on observation-action trajectories. The resulting action-conditioned world model can be used for planning by simulating candidate trajectories in latent space.
Abstract (translated)
我们介绍了DINO-world，这是一种强大的通用视频世界模型，它在DINOv2的潜在空间中训练以预测未来的帧。通过利用一个预训练的图像编码器，并在一个大规模未经策划的视频数据集上训练未来预测器，DINO-world 学习了各种场景中的时间动态变化，包括驾驶、室内场景以及模拟环境等。我们展示了 DINO-world 在多种视频预测基准测试中（如分割和深度预测）优于先前模型，并且在直观物理的理解方面表现出色。此外，我们还表明可以在观察-行动轨迹上对预测器进行微调。由此产生的基于动作的条件世界模型可以通过在潜在空间中模拟候选轨迹来进行规划。
URL
https://arxiv.org/abs/2507.19468
