Abstract
Predicting future video frames is a challenging task with many downstream applications. Previous work has shown that procedural knowledge enables deep models for complex dynamical settings, however their model ViPro assumed a given ground truth initial symbolic state. We show that this approach led to the model learning a shortcut that does not actually connect the observed environment with the predicted symbolic state, resulting in the inability to estimate states given an observation if previous states are noisy. In this work, we add several improvements to ViPro that enables the model to correctly infer states from observations without providing a full ground truth state in the beginning. We show that this is possible in an unsupervised manner, and extend the original Orbits dataset with a 3D variant to close the gap to real world scenarios.
Abstract (translated)
预测未来视频帧是一项具有许多下游应用的挑战性任务。先前的研究表明，程序性知识使深度模型能够在复杂的动态环境中发挥作用，然而它们的模型ViPro假设了一个已知的真实初始符号状态。我们发现这种方法导致了模型学习了一条捷径，这条捷径实际上没有将观察到的环境与预测的符号状态联系起来，结果是无法根据存在噪声的情况下估计出的状态进行预测。 在这项工作中，我们在ViPro的基础上增加了一些改进措施，使模型能够在不提供完整的真实初始状态下正确地从观测中推断状态。我们展示了这种做法可以在无监督的方式下实现，并通过扩展原始的Orbits数据集添加了一个3D变体来缩小与现实世界场景之间的差距。
URL
https://arxiv.org/abs/2508.06335
