Abstract
We investigate the emergence of intuitive physics understanding in general-purpose deep neural network models trained to predict masked regions in natural videos. Leveraging the violation-of-expectation framework, we find that video prediction models trained to predict outcomes in a learned representation space demonstrate an understanding of various intuitive physics properties, such as object permanence and shape consistency. In contrast, video prediction in pixel space and multimodal large language models, which reason through text, achieve performance closer to chance. Our comparisons of these architectures reveal that jointly learning an abstract representation space while predicting missing parts of sensory input, akin to predictive coding, is sufficient to acquire an understanding of intuitive physics, and that even models trained on one week of unique video achieve above chance performance. This challenges the idea that core knowledge -- a set of innate systems to help understand the world -- needs to be hardwired to develop an understanding of intuitive physics.
Abstract (translated)
我们研究了通用深度神经网络模型在训练预测自然视频中被遮挡区域时，直观物理理解的出现。利用违反预期框架，我们发现，在学习表示空间中进行结果预测的视频预测模型展示了对各种直观物理属性的理解，如物体恒常性和形状一致性。相比之下，在像素空间中的视频预测以及通过文本推理的多模态大型语言模型，其表现接近于随机水平。这些架构之间的比较揭示了在预测感觉输入缺失部分的同时共同学习抽象表示空间（类似于预测编码）足以获得对直观物理的理解，并且即使是经过一周独特视频训练的模型也能表现出超过偶然性的性能。这挑战了一种观点，即核心知识——一套帮助理解世界的先天系统——需要被硬连线才能发展出对直观物理的理解。
URL
https://arxiv.org/abs/2502.11831
