Abstract
Predicting future motion trajectories is a critical capability across domains such as robotics, autonomous systems, and human activity forecasting, enabling safer and more intelligent decision-making. This paper proposes a novel, efficient, and lightweight approach for robot action prediction, offering significantly reduced computational cost and inference latency compared to conventional video prediction models. Importantly, it pioneers the adaptation of the InstructPix2Pix model for forecasting future visual frames in robotic tasks, extending its utility beyond static image editing. We implement a deep learning-based visual prediction framework that forecasts what a robot will observe 100 frames (10 seconds) into the future, given a current image and a textual instruction. We repurpose and fine-tune the InstructPix2Pix model to accept both visual and textual inputs, enabling multimodal future frame prediction. Experiments on the RoboTWin dataset (generated based on real-world scenarios) demonstrate that our method achieves superior SSIM and PSNR compared to state-of-the-art baselines in robot action prediction tasks. Unlike conventional video prediction models that require multiple input frames, heavy computation, and slow inference latency, our approach only needs a single image and a text prompt as input. This lightweight design enables faster inference, reduced GPU demands, and flexible multimodal control, particularly valuable for applications like robotics and sports motion trajectory analytics, where motion trajectory precision is prioritized over visual fidelity.
Abstract (translated)
未来运动轨迹的预测在机器人技术、自主系统和人类活动预测等领域中是一项至关重要的能力，它能够支持更安全和智能的决策制定。本文提出了一种新颖、高效且轻量级的方法用于机器人的动作预测，相比传统的视频预测模型，在计算成本和推理延迟方面有显著减少。尤为重要的是，该方法首次将InstructPix2Pix模型应用于机器人任务中的未来视觉帧预测，从而扩展了其在静态图像编辑之外的应用范围。 我们实现了一个基于深度学习的视觉预测框架，能够根据当前的一张图像和一条文本指令来预测100帧（即10秒）后的机器人所观察到的内容。通过重新利用并微调InstructPix2Pix模型以同时接受视觉和文本输入，实现了多模态未来帧预测。 在基于真实场景生成的RoboTWin数据集上的实验表明，我们的方法在机器人的动作预测任务中与现有最佳基线相比，在SSIM（结构相似性指数）和PSNR（峰值信噪比）上表现出更优的结果。不同于传统的视频预测模型需要多个输入帧、计算量大且推理延迟慢的特点，我们提出的方法仅需一张图像和一个文本提示作为输入。 这一轻量级设计不仅实现了更快的推断速度，还减少了对GPU的需求，并提供了灵活多模态控制的可能性，这对于机器人技术以及体育运动轨迹分析等应用场景特别有价值，在这些领域中动作轨迹精度比视觉保真度更为重要。
URL
https://arxiv.org/abs/2507.14809
