Abstract
A unified video and action model holds significant promise for robotics, where videos provide rich scene information for action prediction, and actions provide dynamics information for video prediction. However, effectively combining video generation and action prediction remains challenging, and current video generation-based methods struggle to match the performance of direct policy learning in action accuracy and inference this http URL bridge this gap, we introduce the Unified Video Action model (UVA), which jointly optimizes video and action predictions to achieve both high accuracy and efficient action inference. The key lies in learning a joint video-action latent representation and decoupling video-action decoding. The joint latent representation bridges the visual and action domains, effectively modeling the relationship between video and action sequences. Meanwhile, the decoupled decoding, powered by two lightweight diffusion heads, enables high-speed action inference by bypassing video generation during inference. Such a unified framework further enables versatile functionality through masked input training. By selectively masking actions or videos, a single model can tackle diverse tasks beyond policy learning, such as forward and inverse dynamics modeling and video generation. Via an extensive set of experiments, we demonstrate that UVA can serve as a general-purpose solution for a wide range of robotics tasks, such as policy learning, forward/inverse dynamics and video observation prediction, without compromising performance compared to methods tailored for specific applications. Results are best viewed on this https URL.
Abstract (translated)
一个统一的视频和动作模型在机器人技术中具有巨大的潜力，因为视频提供了丰富的场景信息用于动作预测，而动作则为视频预测提供了动态信息。然而，有效地结合视频生成与动作预测仍然是一项挑战，目前基于视频生成的方法难以在动作准确性及推理效率上与直接策略学习方法相匹敌。为了弥合这一差距，我们引入了统一视频动作模型（UVA），该模型同时优化视频和动作预测，以实现高精度的同时提高行动推断的效率。关键在于学习一个联合的视频-动作潜在表示，并解耦视频-动作解码过程。这种联合潜在表示连接了视觉与行动领域，有效地建模了视频序列与动作序列之间的关系。与此同时，通过两个轻量级扩散头支持的解耦解码，在推断过程中绕过视频生成，使得高速的动作推理成为可能。这样的统一框架还能够通过掩码输入训练实现多功能性。通过对特定的动作或视频进行选择性屏蔽，单一模型可以处理超出策略学习范围的各种任务，例如正向和逆向动力学建模以及视频生成。通过一系列广泛的实验，我们证明了UVA可以作为机器人技术广泛任务（如策略学习、前向/后向动力学及视频观测预测）的通用解决方案，在性能上不逊于专门为特定应用定制的方法。 请注意原文中的“this http URL”是一个链接占位符，可能指向具体的研究成果或实验数据页面。
URL
https://arxiv.org/abs/2503.00200
