Abstract
Video prediction is a crucial task for intelligent agents such as robots and autonomous vehicles, since it enables them to anticipate and act early on time-critical incidents. State-of-the-art video prediction methods typically model the dynamics of a scene jointly and implicitly, without any explicit decomposition into separate objects. This is challenging and potentially sub-optimal, as every object in a dynamic scene has their own pattern of movement, typically somewhat independent of others. In this paper, we investigate the benefit of explicitly modeling the objects in a dynamic scene separately within the context of latent-transformer video prediction models. We conduct detailed and carefully-controlled experiments on both synthetic and real-world datasets; our results show that decomposing a dynamic scene leads to higher quality predictions compared with models of a similar capacity that lack such decomposition.
Abstract (translated)
视频预测对于智能代理（如机器人和自动驾驶车辆）来说是一项关键任务，因为它使它们能够预见并提前应对时间敏感的事件。目前最先进的视频预测方法通常会同时隐式地建模场景的动力学特性，并不明确地将其分解为单独的对象。这种做法具有挑战性且可能次优，因为动态场景中的每个对象都有自己独特的运动模式，这些模式通常是相互独立的。在本文中，我们研究了在潜变量变压器视频预测模型的背景下，显式地分别建模动态场景中的各个对象所带来的益处。我们在合成和真实世界的多个数据集上进行了详细且严格控制的实验；我们的结果显示，在具有类似容量但没有这种分解的模型相比，将动态场景进行分解可以提高预测质量。
URL
https://arxiv.org/abs/2501.10562
