Abstract
Bimanual manipulation is crucial in robotics, enabling complex tasks in industrial automation and household services. However, it poses significant challenges due to the high-dimensional action space and intricate coordination requirements. While video prediction has been recently studied for representation learning and control, leveraging its ability to capture rich dynamic and behavioral information, its potential for enhancing bimanual coordination remains underexplored. To bridge this gap, we propose a unified diffusion-based framework for the joint optimization of video and action prediction. Specifically, we propose a multi-frame latent prediction strategy that encodes future states in a compressed latent space, preserving task-relevant features. Furthermore, we introduce a unidirectional attention mechanism where video prediction is conditioned on the action, while action prediction remains independent of video prediction. This design allows us to omit video prediction during inference, significantly enhancing efficiency. Experiments on two simulated benchmarks and a real-world setting demonstrate a significant improvement in the success rate over the strong baseline ACT using our method, achieving a \textbf{24.9\%} increase on ALOHA, an \textbf{11.1\%} increase on RoboTwin, and a \textbf{32.5\%} increase in real-world experiments. Our models and code are publicly available at this https URL.
Abstract (translated)
双臂操作在机器人技术中至关重要，它能够实现工业自动化和家庭服务中的复杂任务。然而，由于高维的动作空间以及复杂的协调要求，这带来了重大挑战。虽然视频预测最近已被研究用于表示学习和控制，并且可以利用其捕捉丰富动态与行为信息的能力，但该方法在增强双臂协作方面的潜力尚未被充分探索。 为了解决这一问题，我们提出了一种基于扩散的统一框架，旨在联合优化视频和动作预测。具体来说，我们提出了一个多帧潜在预测策略，在这种策略中，未来的状态编码在一个压缩的潜在空间内进行，并保留与任务相关的特征信息。此外，我们还引入了一个单向注意力机制，其中视频预测以行动为条件，而动作预测则独立于视频预测。这一设计允许我们在推理过程中省略视频预测，从而显著提升效率。 在两个模拟基准和一个真实世界设置中的实验表明，我们的方法相较于强大的基线方法ACT，在ALOHA上的成功率提高了**24.9%**, 在RoboTwin上提高了**11.1%**, 并且在现实世界的实验中提升了**32.5%**。我们公开发布了模型和代码，详情请访问[此处](https://this https URL)（请注意，链接应为实际存在的地址）。
URL
https://arxiv.org/abs/2507.11296
