Abstract
Vision-Language-Action (VLA) models have emerged as a promising paradigm for robot learning, but their representations are still largely inherited from static image-text pretraining, leaving physical dynamics to be learned from comparatively limited action data. Generative video models, by contrast, encode rich spatiotemporal structure and implicit physics, making them a compelling foundation for robotic manipulation. But their potentials are not fully explored in the literature. To bridge the gap, we introduce DiT4DiT, an end-to-end Video-Action Model that couples a video Diffusion Transformer with an action Diffusion Transformer in a unified cascaded framework. Instead of relying on reconstructed future frames, DiT4DiT extracts intermediate denoising features from the video generation process and uses them as temporally grounded conditions for action prediction. We further propose a dual flow-matching objective with decoupled timesteps and noise scales for video prediction, hidden-state extraction, and action inference, enabling coherent joint training of both modules. Across simulation and real-world benchmarks, DiT4DiT achieves state-of-the-art results, reaching average success rates of 98.6% on LIBERO and 50.8% on RoboCasa GR1 while using substantially less training data. On the Unitree G1 robot, it also delivers superior real-world performance and strong zero-shot generalization. Importantly, DiT4DiT improves sample efficiency by over 10x and speeds up convergence by up to 7x, demonstrating that video generation can serve as an effective scaling proxy for robot policy learning. We release code and models at this https URL.
Abstract (translated)
视觉-语言-行动（VLA）模型作为机器人学习的一个有前景的范式已经出现，但它们的表示方法仍然很大程度上继承自静态图像和文本预训练，使得物理动态只能从相对有限的动作数据中被学习。相比之下，生成视频模型编码了丰富的时空结构以及隐含的物理学原理，这使它们成为操纵机器人的一个非常有吸引力的基础。然而，在文献中这些潜力尚未得到充分探索。 为弥合这一差距，我们引入了DiT4DiT，这是一个端到端的视频-动作模型，它在一个统一级联框架内将视频扩散变换器与行动扩散变换器耦合在一起。不同于依赖于重建的未来帧，DiT4DiT从视频生成过程中提取中间降噪特征，并将其作为时间基础条件用于动作预测。我们还提出了一个解耦的时间步和噪声尺度的双流匹配目标，用于视频预测、隐藏状态提取以及行动推断，从而使两个模块能够进行连贯的联合训练。 在模拟环境和现实世界基准测试中，DiT4DiT取得了最先进的成果，在LIBERO上达到了平均成功率98.6%，在RoboCasa GR1上达到50.8%的成功率，而使用的训练数据却少得多。此外，在Unitree G1机器人上，它也展示了卓越的实际性能和强大的零样本泛化能力。 重要的是，DiT4DiT通过超过10倍的提高样本效率以及最多7倍加快收敛速度，证明了视频生成可以作为有效的比例因子用于机器人的策略学习。我们将在上述网址发布代码和模型。
URL
https://arxiv.org/abs/2603.10448
