Abstract
The scarcity of manipulation data has motivated the use of pretrained large models from other modalities in robotics. In this work, we build upon autoregressive video generation models to propose a Physical Autoregressive Model (PAR), where physical tokens combine frames and actions to represent the joint evolution of the robot and its environment. PAR leverages the world knowledge embedded in video pretraining to understand physical dynamics without requiring action pretraining, enabling accurate video prediction and consistent action trajectories. It also adopts a DiT-based de-tokenizer to model frames and actions as continuous tokens, mitigating quantization errors and facilitating mutual enhancement. Furthermore, we incorporate a causal mask with inverse kinematics, parallel training, and the KV-cache mechanism to further improve performance and efficiency. Experiments on the ManiSkill benchmark show that PAR achieves a 100\% success rate on the PushCube task, matches the performance of action-pretrained baselines on other tasks, and accurately predicts future videos with tightly aligned action trajectories. These findings underscore a promising direction for robotic manipulation by transferring world knowledge from autoregressive video pretraining.
Abstract (translated)
数据操纵的稀缺性促使了在机器人技术中使用其他模式的预训练大型模型。在这项工作中，我们基于自回归视频生成模型提出了一个物理自回归模型（PAR），其中物理标记结合了帧和动作以表示机器人及其环境的共同演化。PAR利用嵌入在视频预训练中的世界知识来理解物理动态，而无需进行动作预训练，从而实现了准确的视频预测和一致的动作轨迹。它还采用基于DiT的去标记化器将帧和动作作为连续令牌建模，减少了量化误差，并促进了相互增强。此外，我们通过因果掩码、逆向运动学、并行训练以及KV缓存机制来进一步提高性能和效率。 在ManiSkill基准上的实验表明，PAR在PushCube任务上实现了100%的成功率，在其他任务中与动作预训练的基线表现相匹配，并且能够准确预测未来视频，同时确保动作轨迹紧密对齐。这些发现强调了通过自回归视频预训练转移世界知识来促进机器人操纵的一个有希望的方向。
URL
https://arxiv.org/abs/2508.09822
