Abstract
We propose Heterogeneous Masked Autoregression (HMA) for modeling action-video dynamics to generate high-quality data and evaluation in scaling robot learning. Building interactive video world models and policies for robotics is difficult due to the challenge of handling diverse settings while maintaining computational efficiency to run in real time. HMA uses heterogeneous pre-training from observations and action sequences across different robotic embodiments, domains, and tasks. HMA uses masked autoregression to generate quantized or soft tokens for video predictions. \ourshort achieves better visual fidelity and controllability than the previous robotic video generation models with 15 times faster speed in the real world. After post-training, this model can be used as a video simulator from low-level action inputs for evaluating policies and generating synthetic data. See this link this https URL for more information.
Abstract (translated)
我们提出了一种异构掩码自回归（HMA）方法，用于建模动作视频动态，并生成高质量的数据和评估以扩展机器人学习。由于需要处理不同设置的多样性并保持实时计算效率，为机器人建立互动视频世界模型和策略具有挑战性。HMA利用来自不同机器人实体、领域和任务中的观测数据和行动序列进行异构预训练。HMA通过掩码自回归生成量化或软令牌用于视频预测。 \ourshort 模型比之前的机器人视频生成模型在视觉真实感和控制能力上都有所提高，且其实时运行速度提高了15倍。经过后期训练后，该模型可以作为从低级动作输入生成合成数据并评估策略的视频模拟器使用。更多详情请参阅此链接：[https URL]。 请注意，原文中的“\ourshort”可能是指代某个特定版本或简写形式，在翻译时保留了其原始格式，如果需要进一步解释，请提供上下文信息以便准确处理。
URL
https://arxiv.org/abs/2502.04296
