Abstract
Inspired by the performance and scalability of autoregressive large language models (LLMs), transformer-based models have seen recent success in the visual domain. This study investigates a transformer adaptation for video prediction with a simple end-to-end approach, comparing various spatiotemporal self-attention layouts. Focusing on causal modeling of physical simulations over time; a common shortcoming of existing video-generative approaches, we attempt to isolate spatiotemporal reasoning via physical object tracking metrics and unsupervised training on physical simulation datasets. We introduce a simple yet effective pure transformer model for autoregressive video prediction, utilizing continuous pixel-space representations for video prediction. Without the need for complex training strategies or latent feature-learning components, our approach significantly extends the time horizon for physically accurate predictions by up to 50% when compared with existing latent-space approaches, while maintaining comparable performance on common video quality metrics. In addition, we conduct interpretability experiments to identify network regions that encode information useful to perform accurate estimations of PDE simulation parameters via probing models, and find that this generalizes to the estimation of out-of-distribution simulation parameters. This work serves as a platform for further attention-based spatiotemporal modeling of videos via a simple, parameter efficient, and interpretable approach.
Abstract (translated)
受到自回归大型语言模型（LLMs）在性能和可扩展性方面的启发，基于Transformer的模型在视觉领域取得了近期的成功。本研究探讨了一种用于视频预测的Transformer适应方案，并采用简单的端到端方法比较了各种时空自我注意布局。专注于对物理仿真进行因果建模，这通常是现有视频生成方法的一个常见不足之处，我们尝试通过物理对象跟踪指标和在物理仿真数据集上的无监督训练来隔离时空推理。 我们引入了一种简单而有效的纯Transformer模型用于自回归视频预测，该模型利用连续的像素空间表示来进行视频预测。无需复杂的训练策略或潜在特征学习组件，我们的方法与现有的潜在空间方法相比，在物理准确性的预测时间范围上提高了50%以上，同时保持了在常见视频质量指标上的相当性能。 此外，我们进行了可解释性实验以识别网络区域，这些区域通过探针模型编码执行PDE（偏微分方程）仿真参数的精确估计所需的信息，并发现这可以推广到对分布外仿真参数的估计。这项工作提供了一个平台，用于进一步基于注意力机制进行视频的时空建模，采用简单、参数高效且可解释的方法。
URL
https://arxiv.org/abs/2510.20807
