Abstract
Predicting future scene representations is a crucial task for enabling robots to understand and interact with the environment. However, most existing methods rely on video sequences and simulations with precise action annotations, limiting their ability to leverage the large amount of available unlabeled video data. To address this challenge, we propose PlaySlot, an object-centric video prediction model that infers object representations and latent actions from unlabeled video sequences. It then uses these representations to forecast future object states and video frames. PlaySlot allows to generate multiple possible futures conditioned on latent actions, which can be inferred from video dynamics, provided by a user, or generated by a learned action policy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlot outperforms both stochastic and object-centric baselines for video prediction across different environments. Furthermore, we show that our inferred latent actions can be used to learn robot behaviors sample-efficiently from unlabeled video demonstrations. Videos and code are available at this https URL.
Abstract (translated)
预测未来场景表示对于使机器人理解和与环境互动至关重要。然而，大多数现有方法依赖于带有精确动作标注的视频序列和模拟数据，这限制了它们利用大量未标记视频数据的能力。为了解决这一挑战，我们提出了PlaySlot，这是一种以物体为中心的视频预测模型，可以从未标记的视频序列中推断出对象表示和潜在的动作。然后，它使用这些表示来预测未来对象状态和视频帧。PlaySlot可以生成基于从视频动态、用户提供的或由学习到的动作策略生成的潜在动作的多个可能的未来场景，从而实现灵活且可解释的世界建模。我们的结果显示，与不同环境下的视频预测基准相比，PlaySlot在随机基线和以物体为中心的模型中都表现出色。此外，我们展示了通过未标记的视频演示，可以利用推断出的潜在动作有效地学习机器人的行为。更多详细信息、视频及代码可在提供的链接中找到。
URL
https://arxiv.org/abs/2502.07600
