Abstract
Video generation models (VGMs) offer a promising pathway for unified world modeling in robotics by integrating simulation, prediction, and manipulation. However, their practical application remains limited due to (1) slowgeneration speed, which limits real-time interaction, and (2) poor consistency between imagined videos and executable actions. To address these challenges, we propose Manipulate in Dream (MinD), a hierarchical diffusion-based world model framework that employs a dual-system design for vision-language manipulation. MinD executes VGM at low frequencies to extract video prediction features, while leveraging a high-frequency diffusion policy for real-time interaction. This architecture enables low-latency, closed-loop control in manipulation with coherent visual guidance. To better coordinate the two systems, we introduce a video-action diffusion matching module (DiffMatcher), with a novel co-training strategy that uses separate schedulers for each diffusion model. Specifically, we introduce a diffusion-forcing mechanism to DiffMatcher that aligns their intermediate representations during training, helping the fast action model better understand video-based predictions. Beyond manipulation, MinD also functions as a world simulator, reliably predicting task success or failure in latent space before execution. Trustworthy analysis further shows that VGMs can preemptively evaluate task feasibility and mitigate risks. Extensive experiments across multiple benchmarks demonstrate that MinD achieves state-of-the-art manipulation (63%+) in RL-Bench, advancing the frontier of unified world modeling in robotics.
Abstract (translated)
视频生成模型（VGM）为机器人统一世界建模提供了一条有前景的路径，通过整合仿真、预测和操作功能。然而，由于以下两个原因，其实际应用仍受到限制：(1) 生成速度慢，这限制了实时互动；(2) 想象中的视频与可执行动作之间的一致性较差。为了解决这些挑战，我们提出了“梦中操控”（MinD），这是一个基于层次化扩散的世界模型框架，采用双系统设计进行视觉-语言操作。MinD以低频执行VGM来提取视频预测特征，同时利用高频扩散策略实现实时互动。这种架构允许在操作过程中通过连贯的视觉引导实现低延迟、闭环控制。 为了更好地协调这两个系统，我们引入了一个视频-动作扩散匹配模块（DiffMatcher），并提出了一种新颖的联合训练策略，该策略为每个扩散模型使用独立的时间表。具体来说，我们在DiffMatcher中引入了扩散强制机制，在训练过程中将它们的中间表示对齐，帮助快速动作模型更好地理解基于视频的预测。 除了操控功能之外，MinD还可以作为一个世界模拟器工作，在执行之前可靠地在潜在空间内预测任务的成功或失败。值得信赖的分析进一步表明，VGM可以预先评估任务可行性并降低风险。 跨多个基准进行的广泛实验显示，MinD在RL-Bench上的操作方面实现了最先进的性能（超过63%），这推进了机器人统一世界建模的前沿领域。
URL
https://arxiv.org/abs/2506.18897
