Video_Prediction
-
A Real-Time Human Action Recognition Model for Assisted Living
2025-03-18 20:22:17Yixuan Wang, Paul Stynes, Pramod Pathak, Cristina MunteanAbstract
Ensuring the safety and well-being of elderly and vulnerable populations in assisted living environments is a critical concern. Computer vision presents an innovative and powerful approach to predicting health risks through video monitoring, employing human action recognition (HAR) technology. However, real-time prediction of human actions with high performance and efficiency is a challenge. This research proposes a real-time human action recognition model that combines a deep learning model and a live video prediction and alert system, in order to predict falls, staggering and chest pain for residents in assisted living. Six thousand RGB video samples from the NTU RGB+D 60 dataset were selected to create a dataset with four classes: Falling, Staggering, Chest Pain, and Normal, with the Normal class comprising 40 daily activities. Transfer learning technique was applied to train four state-of-the-art HAR models on a GPU server, namely, UniFormerV2, TimeSformer, I3D, and SlowFast. Results of the four models are presented in this paper based on class-wise and macro performance metrics, inference efficiency, model complexity and computational costs. TimeSformer is proposed for developing the real-time human action recognition model, leveraging its leading macro F1 score (95.33%), recall (95.49%), and precision (95.19%) along with significantly higher inference throughput compared to the others. This research provides insights to enhance safety and health of the elderly and people with chronic illnesses in assisted living environments, fostering sustainable care, smarter communities and industry innovation.
Abstract (translated)
确保在辅助生活环境中老年人和弱势群体的安全与健康是至关重要的问题。计算机视觉通过视频监控提供了一种创新且强大的方法，利用人体行为识别（HAR）技术来预测健康风险。然而，实现高性能、高效率的实时人体动作预测是一个挑战。本研究提出了一种结合深度学习模型和实时视频预测及警报系统的实时人体行为识别模型，旨在预测辅助生活环境中居民的跌倒、踉跄和胸痛情况。从NTU RGB+D 60数据集中选择了六千个RGB视频样本，构建了一个包含四类（跌倒、踉跄、胸痛和正常）的数据集，其中“正常”类别包括40种日常活动。采用迁移学习技术在GPU服务器上训练了四种最先进的HAR模型：UniFormerV2、TimeSformer、I3D 和 SlowFast。本文根据各类别的性能指标（如F1分数）、宏观性能指标以及推理效率和模型复杂度及计算成本，展示了这四个模型的结果。基于其卓越的宏观F1得分（95.33%）、召回率（95.49%）和准确率（95.19%），并具有显著高于其他模型的推理吞吐量，TimeSformer被提议用于开发实时人体行为识别模型。这项研究为提高辅助生活环境中老年人及慢性病患者的安全与健康提供了见解，并促进了可持续护理、智能社区以及行业创新的发展。
URL
https://arxiv.org/abs/2503.18957
PDF
-
Frame-wise Conditioning Adaptation for Fine-Tuning Diffusion Models in Text-to-Video Prediction
2025-03-17 09:06:21Zheyuan Liu, Junyan Wang, Zicheng Duan, Cristian Rodriguez-Opazo, Anton van den HengelAbstract
Text-video prediction (TVP) is a downstream video generation task that requires a model to produce subsequent video frames given a series of initial video frames and text describing the required motion. In practice TVP methods focus on a particular category of videos depicting manipulations of objects carried out by human beings or robot arms. Previous methods adapt models pre-trained on text-to-image tasks, and thus tend to generate video that lacks the required continuity. A natural progression would be to leverage more recent pre-trained text-to-video (T2V) models. This approach is rendered more challenging by the fact that the most common fine-tuning technique, low-rank adaptation (LoRA), yields undesirable results. In this work, we propose an adaptation-based strategy we label Frame-wise Conditioning Adaptation (FCA). Within the module, we devise a sub-module that produces frame-wise text embeddings from the input text, which acts as an additional text condition to aid generation. We use FCA to fine-tune the T2V model, which incorporates the initial frame(s) as an extra condition. We compare and discuss the more effective strategy for injecting such embeddings into the T2V model. We conduct extensive ablation studies on our design choices with quantitative and qualitative performance analysis. Our approach establishes a new state-of-the-art for the task of TVP. The project page is at this https URL .
Abstract (translated)
文本到视频预测（TVP）是一项下游视频生成任务，要求模型在给定一系列初始视频帧和描述所需运动的文本的情况下，产生后续的视频帧。实践中，TVP 方法主要关注一类特定类型的视频，这些视频展示了由人类或机械臂操作的对象操纵过程。先前的方法通常会调整那些预先训练于文本到图像任务的模型，因而生成的视频往往缺乏所需的连贯性。一种自然的发展方向是利用最近预训练的文本到视频（T2V）模型。然而，这种做法由于最常用的微调技术——低秩适应（LoRA）产生了不理想的结果而变得更加具有挑战性。 在本项工作中，我们提出了一种基于调整策略的方法，称为帧级条件适配（FCA）。在该模块内，我们设计了一个子模块，它能够从输入文本中生成每一帧的文本嵌入，这一额外的文字条件有助于视频生成。我们将使用FCA对T2V模型进行微调，并将其初始帧作为附加条件纳入其中。我们会比较和讨论将此类嵌入注入T2V模型中的更有效策略。我们进行了广泛的消融研究来评估设计选择的定量和定性性能。 我们的方法为TVP任务建立了新的最先进水平，项目页面在此[URL]。
URL
https://arxiv.org/abs/2503.12953
PDF
-
GMG: A Video Prediction Method Based on Global Focus and Motion Guided
2025-03-14 11:06:49Yuhao Du, Hui Liu, Haoxiang Peng, Xinyuan Chen, Chenrong Wu, Jiankai ZhangAbstract
Recent years, weather forecasting has gained significant attention. However, accurately predicting weather remains a challenge due to the rapid variability of meteorological data and potential teleconnections. Current spatiotemporal forecasting models primarily rely on convolution operations or sliding windows for feature extraction. These methods are limited by the size of the convolutional kernel or sliding window, making it difficult to capture and identify potential teleconnection features in meteorological data. Additionally, weather data often involve non-rigid bodies, whose motion processes are accompanied by unpredictable deformations, further complicating the forecasting task. In this paper, we propose the GMG model to address these two core challenges. The Global Focus Module, a key component of our model, enhances the global receptive field, while the Motion Guided Module adapts to the growth or dissipation processes of non-rigid bodies. Through extensive evaluations, our method demonstrates competitive performance across various complex tasks, providing a novel approach to improving the predictive accuracy of complex spatiotemporal data.
Abstract (translated)
近年来，天气预报受到了越来越多的关注。然而，由于气象数据的快速变化和潜在的远程联系（teleconnections），准确预测天气仍然是一个挑战。目前的空间时间预测模型主要依赖于卷积操作或滑动窗口来进行特征提取。这些方法受到卷积核大小或滑动窗口大小的限制，难以捕捉并识别气象数据中的潜在远程联系特征。此外，天气数据通常涉及非刚体物体，其运动过程伴随着不可预知的变形，进一步增加了预测任务的复杂性。 在本文中，我们提出了GMG模型来解决上述两个核心挑战。该模型的关键组成部分是全局关注模块（Global Focus Module），它增强了全局感受野；另一个重要部分是运动引导模块（Motion Guided Module），它可以适应非刚体物体的成长或消散过程。通过广泛的评估，我们的方法在各种复杂任务中展示了竞争性的性能，为提高复杂空间时间数据的预测准确性提供了一种新颖的方法。
URL
https://arxiv.org/abs/2503.11297
PDF
-
Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learning
2025-03-11 13:50:22Qi Wang, Zhipeng Zhang, Baao Xie, Xin Jin, Yunbo Wang, Shiyu Wang, Liaomo Zheng, Xiaokang Yang, Wenjun ZengAbstract
Training visual reinforcement learning (RL) in practical scenarios presents a significant challenge, $\textit{i.e.,}$ RL agents suffer from low sample efficiency in environments with variations. While various approaches have attempted to alleviate this issue by disentanglement representation learning, these methods usually start learning from scratch without prior knowledge of the world. This paper, in contrast, tries to learn and understand underlying semantic variations from distracting videos via offline-to-online latent distillation and flexible disentanglement constraints. To enable effective cross-domain semantic knowledge transfer, we introduce an interpretable model-based RL framework, dubbed Disentangled World Models (DisWM). Specifically, we pretrain the action-free video prediction model offline with disentanglement regularization to extract semantic knowledge from distracting videos. The disentanglement capability of the pretrained model is then transferred to the world model through latent distillation. For finetuning in the online environment, we exploit the knowledge from the pretrained model and introduce a disentanglement constraint to the world model. During the adaptation phase, the incorporation of actions and rewards from online environment interactions enriches the diversity of the data, which in turn strengthens the disentangled representation learning. Experimental results validate the superiority of our approach on various benchmarks.
Abstract (translated)
在实际场景中训练视觉强化学习（RL）面临重大挑战，即在环境变化的情况下，RL代理的样本效率低下。尽管有许多方法尝试通过解耦表示学习来缓解这一问题，这些方法通常从零开始学习，并不使用世界知识。相比之下，本文试图通过离线到在线潜在蒸馏和灵活的解耦约束从分散视频中学习并理解底层语义变化。为了实现有效的跨域语义知识转移，我们引入了一个可解释的基于模型的RL框架，称为解耦世界模型（DisWM）。具体来说，我们在离线状态下使用带有解耦正则化的无动作视频预测模型进行预训练，以从分散视频中提取语义知识。接着，通过潜在蒸馏将预训练模型的解耦能力转移到世界模型上。在线环境中微调时，我们利用了预训练模型的知识，并向世界模型引入了解耦约束。在适应阶段，结合在线环境交互中的动作和奖励数据丰富了数据多样性，进一步增强了解耦表示学习。实验结果验证了我们的方法在多个基准测试上的优越性。
URL
https://arxiv.org/abs/2503.08751
PDF
-
Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Models
2025-03-02 16:10:32Xingzhuo Guo, Yu Zhang, Baixu Chen, Haoran Xu, Jianmin Wang, Mingsheng LongAbstract
Diffusion models have emerged as powerful generative frameworks by progressively adding noise to data through a forward process and then reversing this process to generate realistic samples. While these models have achieved strong performance across various tasks and modalities, their application to temporal predictive learning remains underexplored. Existing approaches treat predictive learning as a conditional generation problem, but often fail to fully exploit the temporal dynamics inherent in the data, leading to challenges in generating temporally coherent sequences. To address this, we introduce Dynamical Diffusion (DyDiff), a theoretically sound framework that incorporates temporally aware forward and reverse processes. Dynamical Diffusion explicitly models temporal transitions at each diffusion step, establishing dependencies on preceding states to better capture temporal dynamics. Through the reparameterization trick, Dynamical Diffusion achieves efficient training and inference similar to any standard diffusion model. Extensive experiments across scientific spatiotemporal forecasting, video prediction, and time series forecasting demonstrate that Dynamical Diffusion consistently improves performance in temporal predictive tasks, filling a crucial gap in existing methodologies. Code is available at this repository: this https URL.
Abstract (translated)
扩散模型作为一种强大的生成框架，通过逐步向数据添加噪声的正向过程和反向该过程以生成逼真的样本而兴起。尽管这些模型在各种任务和模态中表现出色，但它们在时间预测学习中的应用尚未得到充分探索。现有方法将预测学习视为条件生成问题，但由于未能充分利用数据内在的时间动态特性，通常难以生成具有连贯性的时间序列。为解决这一问题，我们引入了动力学扩散（DyDiff），这是一种理论依据坚实的框架，它包含了时间感知的正向和反向过程。动力学扩散在每次扩散步骤中显式建模时间转换，并基于先前的状态建立依赖关系，以更好地捕捉时间动态特性。通过重新参数化技巧，动力学扩散实现了类似标准扩散模型的有效训练和推理效率。跨科学时空预测、视频预测以及时间序列预测的广泛实验表明，动力学扩散在时间预测任务中始终提高了性能，填补了现有方法论的一个重要空白。代码可在以下仓库获取：此 https URL 链接。
URL
https://arxiv.org/abs/2503.00951
PDF
-
Unified Video Action Model
2025-02-28 21:38:17Shuang Li, Yihuai Gao, Dorsa Sadigh, Shuran SongAbstract
A unified video and action model holds significant promise for robotics, where videos provide rich scene information for action prediction, and actions provide dynamics information for video prediction. However, effectively combining video generation and action prediction remains challenging, and current video generation-based methods struggle to match the performance of direct policy learning in action accuracy and inference this http URL bridge this gap, we introduce the Unified Video Action model (UVA), which jointly optimizes video and action predictions to achieve both high accuracy and efficient action inference. The key lies in learning a joint video-action latent representation and decoupling video-action decoding. The joint latent representation bridges the visual and action domains, effectively modeling the relationship between video and action sequences. Meanwhile, the decoupled decoding, powered by two lightweight diffusion heads, enables high-speed action inference by bypassing video generation during inference. Such a unified framework further enables versatile functionality through masked input training. By selectively masking actions or videos, a single model can tackle diverse tasks beyond policy learning, such as forward and inverse dynamics modeling and video generation. Via an extensive set of experiments, we demonstrate that UVA can serve as a general-purpose solution for a wide range of robotics tasks, such as policy learning, forward/inverse dynamics and video observation prediction, without compromising performance compared to methods tailored for specific applications. Results are best viewed on this https URL.
Abstract (translated)
一个统一的视频和动作模型在机器人技术中具有巨大的潜力，因为视频提供了丰富的场景信息用于动作预测，而动作则为视频预测提供了动态信息。然而，有效地结合视频生成与动作预测仍然是一项挑战，目前基于视频生成的方法难以在动作准确性及推理效率上与直接策略学习方法相匹敌。为了弥合这一差距，我们引入了统一视频动作模型（UVA），该模型同时优化视频和动作预测，以实现高精度的同时提高行动推断的效率。关键在于学习一个联合的视频-动作潜在表示，并解耦视频-动作解码过程。这种联合潜在表示连接了视觉与行动领域，有效地建模了视频序列与动作序列之间的关系。与此同时，通过两个轻量级扩散头支持的解耦解码，在推断过程中绕过视频生成，使得高速的动作推理成为可能。这样的统一框架还能够通过掩码输入训练实现多功能性。通过对特定的动作或视频进行选择性屏蔽，单一模型可以处理超出策略学习范围的各种任务，例如正向和逆向动力学建模以及视频生成。通过一系列广泛的实验，我们证明了UVA可以作为机器人技术广泛任务（如策略学习、前向/后向动力学及视频观测预测）的通用解决方案，在性能上不逊于专门为特定应用定制的方法。 请注意原文中的“this http URL”是一个链接占位符，可能指向具体的研究成果或实验数据页面。
URL
https://arxiv.org/abs/2503.00200
PDF
-
Intuitive physics understanding emerges from self-supervised pretraining on natural videos
2025-02-17 14:27:14Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCunAbstract
We investigate the emergence of intuitive physics understanding in general-purpose deep neural network models trained to predict masked regions in natural videos. Leveraging the violation-of-expectation framework, we find that video prediction models trained to predict outcomes in a learned representation space demonstrate an understanding of various intuitive physics properties, such as object permanence and shape consistency. In contrast, video prediction in pixel space and multimodal large language models, which reason through text, achieve performance closer to chance. Our comparisons of these architectures reveal that jointly learning an abstract representation space while predicting missing parts of sensory input, akin to predictive coding, is sufficient to acquire an understanding of intuitive physics, and that even models trained on one week of unique video achieve above chance performance. This challenges the idea that core knowledge -- a set of innate systems to help understand the world -- needs to be hardwired to develop an understanding of intuitive physics.
Abstract (translated)
我们研究了通用深度神经网络模型在训练预测自然视频中被遮挡区域时，直观物理理解的出现。利用违反预期框架，我们发现，在学习表示空间中进行结果预测的视频预测模型展示了对各种直观物理属性的理解，如物体恒常性和形状一致性。相比之下，在像素空间中的视频预测以及通过文本推理的多模态大型语言模型，其表现接近于随机水平。这些架构之间的比较揭示了在预测感觉输入缺失部分的同时共同学习抽象表示空间（类似于预测编码）足以获得对直观物理的理解，并且即使是经过一周独特视频训练的模型也能表现出超过偶然性的性能。这挑战了一种观点，即核心知识——一套帮助理解世界的先天系统——需要被硬连线才能发展出对直观物理的理解。
URL
https://arxiv.org/abs/2502.11831
PDF
-
MaskGWM: A Generalizable Driving World Model with Video Mask Reconstruction
2025-02-17 10:53:56Jingcheng Ni, Yuxin Guo, Yichen Liu, Rui Chen, Lewei Lu, Zehuan WuAbstract
World models that forecast environmental changes from actions are vital for autonomous driving models with strong generalization. The prevailing driving world model mainly build on video prediction model. Although these models can produce high-fidelity video sequences with advanced diffusion-based generator, they are constrained by their predictive duration and overall generalization capabilities. In this paper, we explore to solve this problem by combining generation loss with MAE-style feature-level context learning. In particular, we instantiate this target with three key design: (1) A more scalable Diffusion Transformer (DiT) structure trained with extra mask construction task. (2) we devise diffusion-related mask tokens to deal with the fuzzy relations between mask reconstruction and generative diffusion process. (3) we extend mask construction task to spatial-temporal domain by utilizing row-wise mask for shifted self-attention rather than masked self-attention in MAE. Then, we adopt a row-wise cross-view module to align with this mask design. Based on above improvement, we propose MaskGWM: a Generalizable driving World Model embodied with Video Mask reconstruction. Our model contains two variants: MaskGWM-long, focusing on long-horizon prediction, and MaskGWM-mview, dedicated to multi-view generation. Comprehensive experiments on standard benchmarks validate the effectiveness of the proposed method, which contain normal validation of Nuscene dataset, long-horizon rollout of OpenDV-2K dataset and zero-shot validation of Waymo dataset. Quantitative metrics on these datasets show our method notably improving state-of-the-art driving world model.
Abstract (translated)
世界模型，用于从行动预测环境变化，在具有强泛化能力的自动驾驶模型中至关重要。当前主流的驾驶世界模型主要建立在视频预测模型基础上。尽管这些模型能够利用先进的扩散生成器产生高保真度的视频序列，但它们受制于预测时长和整体泛化的局限性。本文探索通过结合生成损失与类似MAE（掩码图像建模）特征级上下文学习的方法来解决这一问题。 具体而言，我们采用以下三个关键设计来实现这一目标： 1. 一个更可扩展的Diffusion Transformer (DiT) 结构，并且在训练过程中加入了额外的掩码构建任务。 2. 设计了与扩散过程相关的掩码令牌以处理掩码重建和生成式扩散过程之间的模糊关系。 3. 将掩码构建任务扩展到时空域，通过使用行级掩码进行移位自我注意机制（而不是MAE中的屏蔽自我注意），并采用行级跨视图模块来与此设计对齐。 基于上述改进，我们提出了MaskGWM：一个具备视频掩码重建的可泛化驾驶世界模型。该模型包含两种变体： - MaskGWM-long: 专注于长时预测。 - MaskGWM-mview: 面向多视角生成。 在标准基准上的全面实验验证了所提方法的有效性，包括Nuscene数据集的标准验证、OpenDV-2K数据集的长时滚动和Waymo数据集的零样本验证。这些数据集上的定量指标显示我们的方法显著提升了最先进的驾驶世界模型的表现。
URL
https://arxiv.org/abs/2502.11663
PDF
-
Skillful Nowcasting of Convective Clouds With a Cascade Diffusion Model
2025-02-16 02:29:13Haoming Chen, Xiaohui Zhong, Qiang Zhai, Xiaomeng Li, Ying Wa Chan, Pak Wai Chan, Yuanyuan Huang, Hao Li, Xiaoming ShiAbstract
Accurate nowcasting of convective clouds from satellite imagery is essential for mitigating the impacts of meteorological disasters, especially in developing countries and remote regions with limited ground-based observations. Recent advances in deep learning have shown promise in video prediction; however, existing models frequently produce blurry results and exhibit reduced accuracy when forecasting physical fields. Here, we introduce SATcast, a diffusion model that leverages a cascade architecture and multimodal inputs for nowcasting cloud fields in satellite imagery. SATcast incorporates physical fields predicted by FuXi, a deep-learning weather model, alongside past satellite observations as conditional inputs to generate high-quality future cloud fields. Through comprehensive evaluation, SATcast outperforms conventional methods on multiple metrics, demonstrating its superior accuracy and robustness. Ablation studies underscore the importance of its multimodal design and the cascade architecture in achieving reliable predictions. Notably, SATcast maintains predictive skill for up to 24 hours, underscoring its potential for operational nowcasting applications.
Abstract (translated)
从卫星图像准确预测对流云对于减轻气象灾害的影响至关重要，尤其是在地面观测有限的发展中国家和偏远地区。最近在深度学习领域的进展显示出视频预测方面的巨大潜力；然而，现有的模型常常产生模糊的结果，并且在预报物理场时准确性降低。在这里，我们介绍了SATcast，这是一种扩散模型，采用了级联架构并利用多模态输入来对卫星图像中的云层进行临近预报。SATcast结合了深度学习气象模型FuXi预测的物理场以及过去的卫星观测数据作为条件输入，以生成高质量的未来云图。通过全面评估，SATcast在多个指标上优于传统方法，展示了其卓越的准确性和鲁棒性。消融实验强调了其多模态设计和级联架构对于实现可靠预测的重要性。值得注意的是，SATcast能够保持长达24小时的预报技能，突显了它在操作性临近预报中的潜在应用价值。
URL
https://arxiv.org/abs/2502.10957
PDF
-
PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
2025-02-11 14:50:10Angel Villar-Corrales, Sven BehnkeAbstract
Predicting future scene representations is a crucial task for enabling robots to understand and interact with the environment. However, most existing methods rely on video sequences and simulations with precise action annotations, limiting their ability to leverage the large amount of available unlabeled video data. To address this challenge, we propose PlaySlot, an object-centric video prediction model that infers object representations and latent actions from unlabeled video sequences. It then uses these representations to forecast future object states and video frames. PlaySlot allows to generate multiple possible futures conditioned on latent actions, which can be inferred from video dynamics, provided by a user, or generated by a learned action policy, thus enabling versatile and interpretable world modeling. Our results show that PlaySlot outperforms both stochastic and object-centric baselines for video prediction across different environments. Furthermore, we show that our inferred latent actions can be used to learn robot behaviors sample-efficiently from unlabeled video demonstrations. Videos and code are available at this https URL.
Abstract (translated)
预测未来场景表示对于使机器人理解和与环境互动至关重要。然而，大多数现有方法依赖于带有精确动作标注的视频序列和模拟数据，这限制了它们利用大量未标记视频数据的能力。为了解决这一挑战，我们提出了PlaySlot，这是一种以物体为中心的视频预测模型，可以从未标记的视频序列中推断出对象表示和潜在的动作。然后，它使用这些表示来预测未来对象状态和视频帧。PlaySlot可以生成基于从视频动态、用户提供的或由学习到的动作策略生成的潜在动作的多个可能的未来场景，从而实现灵活且可解释的世界建模。我们的结果显示，与不同环境下的视频预测基准相比，PlaySlot在随机基线和以物体为中心的模型中都表现出色。此外，我们展示了通过未标记的视频演示，可以利用推断出的潜在动作有效地学习机器人的行为。更多详细信息、视频及代码可在提供的链接中找到。
URL
https://arxiv.org/abs/2502.07600
PDF
-
Learning Real-World Action-Video Dynamics with Heterogeneous Masked Autoregression
2025-02-06 18:38:26Lirui Wang, Kevin Zhao, Chaoqi Liu, Xinlei ChenAbstract
We propose Heterogeneous Masked Autoregression (HMA) for modeling action-video dynamics to generate high-quality data and evaluation in scaling robot learning. Building interactive video world models and policies for robotics is difficult due to the challenge of handling diverse settings while maintaining computational efficiency to run in real time. HMA uses heterogeneous pre-training from observations and action sequences across different robotic embodiments, domains, and tasks. HMA uses masked autoregression to generate quantized or soft tokens for video predictions. \ourshort achieves better visual fidelity and controllability than the previous robotic video generation models with 15 times faster speed in the real world. After post-training, this model can be used as a video simulator from low-level action inputs for evaluating policies and generating synthetic data. See this link this https URL for more information.
Abstract (translated)
我们提出了一种异构掩码自回归（HMA）方法，用于建模动作视频动态，并生成高质量的数据和评估以扩展机器人学习。由于需要处理不同设置的多样性并保持实时计算效率，为机器人建立互动视频世界模型和策略具有挑战性。HMA利用来自不同机器人实体、领域和任务中的观测数据和行动序列进行异构预训练。HMA通过掩码自回归生成量化或软令牌用于视频预测。 \ourshort 模型比之前的机器人视频生成模型在视觉真实感和控制能力上都有所提高，且其实时运行速度提高了15倍。经过后期训练后，该模型可以作为从低级动作输入生成合成数据并评估策略的视频模拟器使用。更多详情请参阅此链接：[https URL]。 请注意，原文中的“\ourshort”可能是指代某个特定版本或简写形式，在翻译时保留了其原始格式，如果需要进一步解释，请提供上下文信息以便准确处理。
URL
https://arxiv.org/abs/2502.04296
PDF
-
MAUCell: An Adaptive Multi-Attention Framework for Video Frame Prediction
2025-01-28 14:52:10Shreyam Gupta (Indian Institute of Technology), P. Agrawal (University of Colorado, Boulder, USA), Priyam Gupta (Intelligent Field Robotic Systems)Abstract
Temporal sequence modeling stands as the fundamental foundation for video prediction systems and real-time forecasting operations as well as anomaly detection applications. The achievement of accurate predictions through efficient resource consumption remains an ongoing issue in contemporary temporal sequence modeling. We introduce the Multi-Attention Unit (MAUCell) which combines Generative Adversarial Networks (GANs) and spatio-temporal attention mechanisms to improve video frame prediction capabilities. Our approach implements three types of attention models to capture intricate motion sequences. A dynamic combination of these attention outputs allows the model to reach both advanced decision accuracy along with superior quality while remaining computationally efficient. The integration of GAN elements makes generated frames appear more true to life therefore the framework creates output sequences which mimic real-world footage. The new design system maintains equilibrium between temporal continuity and spatial accuracy to deliver reliable video prediction. Through a comprehensive evaluation methodology which merged the perceptual LPIPS measurement together with classic tests MSE, MAE, SSIM and PSNR exhibited enhancing capabilities than contemporary approaches based on direct benchmark tests of Moving MNIST, KTH Action, and CASIA-B (Preprocessed) datasets. Our examination indicates that MAUCell shows promise for operational time requirements. The research findings demonstrate how GANs work best with attention mechanisms to create better applications for predicting video sequences.
Abstract (translated)
时间序列建模是视频预测系统、实时预测操作及异常检测应用的基础。在当代时间序列建模中，如何通过高效利用资源实现准确预测仍是一个亟待解决的问题。我们提出了一种结合生成对抗网络（GANs）和时空注意力机制的多注意单元模型（MAUCell），以提升视频帧预测能力。本方法实现了三种类型的注意力模型来捕捉复杂的运动序列。动态组合这些注意力输出使模型能够在保持计算效率的同时，达到高精度决策和高质量结果。引入GAN元素使得生成的帧看起来更逼真，从而使框架能够创建类似于现实世界的输出序列。新的设计系统在时间连续性和空间准确性之间保持平衡，以提供可靠的视频预测。通过结合感知LPIPS测量与经典测试（MSE、MAE、SSIM和PSNR）的全面评估方法表明，在Moving MNIST、KTH Action 和 CASIA-B（预处理）数据集上进行基准测试时，我们的方法比现有方法表现更优。研究表明，MAUCell在运行时间要求上有潜力实现改进。研究结果展示了GANs与注意力机制结合使用以创建更好的视频序列预测应用的优越性。
URL
https://arxiv.org/abs/2501.16997
PDF
-
Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networks
2025-01-23 02:18:13Mars Liyao Gao, Jan P. Williams, J. Nathan KutzAbstract
Spatiotemporal modeling of real-world data poses a challenging problem due to inherent high dimensionality, measurement noise, and expensive data collection procedures. In this paper, we present Sparse Identification of Nonlinear Dynamics with SHallow REcurrent Decoder networks (SINDy-SHRED), a method to jointly solve the sensing and model identification problems with simple implementation, efficient computation, and robust performance. SINDy-SHRED uses Gated Recurrent Units (GRUs) to model the temporal sequence of sensor measurements along with a shallow decoder network to reconstruct the full spatiotemporal field from the latent state space using only a few available sensors. Our proposed algorithm introduces a SINDy-based regularization; beginning with an arbitrary latent state space, the dynamics of the latent space progressively converges to a SINDy-class functional, provided the projection remains within the set. In restricting SINDy to a linear model, the architecture produces a Koopman-SHRED model which enforces a linear latent space dynamics. We conduct a systematic experimental study including synthetic PDE data, real-world sensor measurements for sea surface temperature, and direct video data. With no explicit encoder, SINDy-SHRED and Koopman-SHRED enable efficient training with minimal hyperparameter tuning and laptop-level computing; further, it demonstrates robust generalization in a variety of applications with minimal to no hyperparameter adjustments. Finally, the interpretable SINDy and Koopman models of latent state dynamics enables accurate long-term video predictions, achieving state-of-the-art performance and outperforming all baseline methods considered, including Convolutional LSTM, PredRNN, ResNet, and SimVP.
Abstract (translated)
现实世界数据的时空建模面临挑战，原因在于其固有的高维度、测量噪声以及昂贵的数据采集过程。本文介绍了一种名为“基于稀疏非线性动力学识别与浅层递归解码器网络的方法”（Sparse Identification of Nonlinear Dynamics with SHallow REcurrent Decoder networks, SINDy-SHRED）的新方法，该方法能够同时解决传感和模型识别问题，并且具有简单实现、高效计算以及鲁棒性能的特点。SINDy-SHRED使用门控循环单元（Gated Recurrent Units, GRUs）来建模传感器测量的时间序列，并结合浅层解码器网络从仅有少量可用传感器的状态空间中重建完整的时空场。 该方法引入了基于SINDy的正则化技术：从任意初始状态空间开始，通过逐步调整动态过程，最终使得隐含空间的动力学逐渐收敛于一类SINDy功能。在将SINDy限制为线性模型的情况下，架构生成一个Koopman-SHRED模型，该模型强制执行线性的隐藏空间动力学。 我们进行了一系列系统实验研究，包括合成PDE数据、海面温度的真实传感器测量以及直接视频数据。没有显式编码器的情况下，SINDy-SHRED和Koopman-SHRED方法允许使用极小的超参数调整和笔记本级别计算实现高效的训练；此外，在各种应用中展示了优秀的泛化能力，并且在这些应用中几乎没有进行任何或很少的超参数调整。 最后，通过可解释的SINDy及Koopman模型对隐含状态动态的建模，实现了准确的长时间视频预测，在性能上超越了所有考虑中的基线方法（包括卷积LSTM、PredRNN、ResNet和SimVP），达到了最先进的水平。
URL
https://arxiv.org/abs/2501.13329
PDF
-
Taming Teacher Forcing for Masked Autoregressive Video Generation
2025-01-21 18:59:31Deyu Zhou, Quan Sun, Yuang Peng, Kun Yan, Runpei Dong, Duomin Wang, Zheng Ge, Nan Duan, Xiangyu Zhang, Lionel M. Ni, Heung-Yeung ShumAbstract
We introduce MAGI, a hybrid video generation framework that combines masked modeling for intra-frame generation with causal modeling for next-frame generation. Our key innovation, Complete Teacher Forcing (CTF), conditions masked frames on complete observation frames rather than masked ones (namely Masked Teacher Forcing, MTF), enabling a smooth transition from token-level (patch-level) to frame-level autoregressive generation. CTF significantly outperforms MTF, achieving a +23% improvement in FVD scores on first-frame conditioned video prediction. To address issues like exposure bias, we employ targeted training strategies, setting a new benchmark in autoregressive video generation. Experiments show that MAGI can generate long, coherent video sequences exceeding 100 frames, even when trained on as few as 16 frames, highlighting its potential for scalable, high-quality video generation.
Abstract (translated)
我们介绍了MAGI，这是一个结合了掩码建模（用于帧内生成）和因果建模（用于下一帧生成）的混合视频生成框架。我们的关键创新是“完整教师强制”（Complete Teacher Forcing，CTF），它使掩码帧依赖于完整的观察帧而非仅依赖于其他掩码帧（即Masked Teacher Forcing，MTF）。这种方法使得从标记级（补丁级）到帧级的自回归生成能够平滑过渡。实验结果显示，CTF显著优于MTF，在第一帧条件下的视频预测中，FVD评分提高了23%。 为了应对诸如曝光偏差等问题，我们采用了有针对性的训练策略，并在自回归视频生成领域设立了新的基准标准。实验表明，MAGI能够在仅使用16帧进行训练的情况下，生成超过100帧长且连贯的视频序列，这突显了其在大规模、高质量视频生成中的巨大潜力。
URL
https://arxiv.org/abs/2501.12389
PDF
-
On the Benefits of Instance Decomposition in Video Prediction Models
2025-01-17 21:36:06Eliyas Suleyman, Paul Henderson, Nicolas PugeaultAbstract
Video prediction is a crucial task for intelligent agents such as robots and autonomous vehicles, since it enables them to anticipate and act early on time-critical incidents. State-of-the-art video prediction methods typically model the dynamics of a scene jointly and implicitly, without any explicit decomposition into separate objects. This is challenging and potentially sub-optimal, as every object in a dynamic scene has their own pattern of movement, typically somewhat independent of others. In this paper, we investigate the benefit of explicitly modeling the objects in a dynamic scene separately within the context of latent-transformer video prediction models. We conduct detailed and carefully-controlled experiments on both synthetic and real-world datasets; our results show that decomposing a dynamic scene leads to higher quality predictions compared with models of a similar capacity that lack such decomposition.
Abstract (translated)
视频预测对于智能代理（如机器人和自动驾驶车辆）来说是一项关键任务，因为它使它们能够预见并提前应对时间敏感的事件。目前最先进的视频预测方法通常会同时隐式地建模场景的动力学特性，并不明确地将其分解为单独的对象。这种做法具有挑战性且可能次优，因为动态场景中的每个对象都有自己独特的运动模式，这些模式通常是相互独立的。在本文中，我们研究了在潜变量变压器视频预测模型的背景下，显式地分别建模动态场景中的各个对象所带来的益处。我们在合成和真实世界的多个数据集上进行了详细且严格控制的实验；我们的结果显示，在具有类似容量但没有这种分解的模型相比，将动态场景进行分解可以提高预测质量。
URL
https://arxiv.org/abs/2501.10562
PDF
-
Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
2024-12-19 12:48:40Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, Jianyu ChenAbstract
Recent advancements in robotics have focused on developing generalist policies capable of performing multiple tasks. Typically, these policies utilize pre-trained vision encoders to capture crucial information from current observations. However, previous vision encoders, which trained on two-image contrastive learning or single-image reconstruction, can not perfectly capture the sequential information essential for embodied tasks. Recently, video diffusion models (VDMs) have demonstrated the capability to accurately predict future image sequences, exhibiting a good understanding of physical dynamics. Motivated by the strong visual prediction capabilities of VDMs, we hypothesize that they inherently possess visual representations that reflect the evolution of the physical world, which we term predictive visual representations. Building on this hypothesis, we propose the Video Prediction Policy (VPP), a generalist robotic policy conditioned on the predictive visual representations from VDMs. To further enhance these representations, we incorporate diverse human or robotic manipulation datasets, employing unified video-generation training objectives. VPP consistently outperforms existing methods across two simulated and two real-world benchmarks. Notably, it achieves a 28.1\% relative improvement in the Calvin ABC-D benchmark compared to the previous state-of-the-art and delivers a 28.8\% increase in success rates for complex real-world dexterous manipulation tasks.
Abstract (translated)
最近的机器人技术进步集中在开发能够执行多种任务的通用策略上。通常，这些策略使用预训练的视觉编码器来捕捉当前观察中的关键信息。然而，先前的视觉编码器是基于两图像对比学习或单图像重建进行训练的，它们无法完美地捕获对具身任务至关重要的顺序信息。最近，视频扩散模型（VDMs）展示了准确预测未来图像序列的能力，表明了其对物理动力学的良好理解。受到 VDM 强大的视觉预测能力的启发，我们假设它们内在拥有反映物理世界演化的视觉表示，我们将这种表示称为预测性视觉表示。基于这一假设，我们提出了视频预测策略（VPP），这是一种根据来自 VDM 的预测性视觉表示条件的通用机器人策略。为了进一步增强这些表示，我们结合了多样的人类或机器人操作数据集，并采用了统一的视频生成训练目标。在两个模拟和两个现实世界的基准测试中，VPP 一直优于现有的方法。值得注意的是，在 Calvin ABC-D 基准上，与之前的最先进技术相比，它实现了相对提高28.1%，并且对于复杂的实际灵巧操控任务的成功率提高了28.8%。
URL
https://arxiv.org/abs/2412.14803
PDF
-
STIV: Scalable Text and Image Conditioned Video Generation
2024-12-10 18:27:06Zongyu Lin, Wei Liu, Chen Chen, Jiasen Lu, Wenze Hu, Tsu-Jui Fu, Jesse Allardice, Zhengfeng Lai, Liangchen Song, Bowen Zhang, Cha Chen, Yiran Fei, Yifan Jiang, Lezhi Li, Yizhou Sun, Kai-Wei Chang, Yinfei YangAbstract
The field of video generation has made remarkable advancements, yet there remains a pressing need for a clear, systematic recipe that can guide the development of robust and scalable models. In this work, we present a comprehensive study that systematically explores the interplay of model architectures, training recipes, and data curation strategies, culminating in a simple and scalable text-image-conditioned video generation method, named STIV. Our framework integrates image condition into a Diffusion Transformer (DiT) through frame replacement, while incorporating text conditioning via a joint image-text conditional classifier-free guidance. This design enables STIV to perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks simultaneously. Additionally, STIV can be easily extended to various applications, such as video prediction, frame interpolation, multi-view generation, and long video generation, etc. With comprehensive ablation studies on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V, surpassing both leading open and closed-source models like CogVideoX-5B, Pika, Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and extensible recipe for building cutting-edge video generation models, we aim to empower future research and accelerate progress toward more versatile and reliable video generation solutions.
Abstract (translated)
视频生成领域取得了显著的进步，但仍迫切需要一个清晰、系统的方案来指导构建强大且可扩展的模型。在这项工作中，我们进行了一项全面的研究，系统地探讨了模型架构、训练方法和数据整理策略之间的相互作用，最终形成了一种简单而可扩展的文本-图像条件视频生成方法，名为STIV。我们的框架通过帧替换将图像条件整合到扩散转换器（DiT）中，并通过联合的图像-文本条件无引导分类器引入文本条件。这种设计使STIV能够同时执行从文本到视频（T2V）和从文本-图像到视频（TI2V）的任务。此外，STIV可以轻松扩展到各种应用，例如视频预测、帧插值、多视图生成和长视频生成等。通过对T2I、T2V和TI2V进行详尽的消融研究，尽管设计简单，STIV仍表现出色。一个87亿参数模型在512分辨率下，在VBench T2V任务上达到了83.1分，超过了领先的开源和闭源模型如CogVideoX-5B、Pika、Kling和Gen-3等。相同规模的模型在512分辨率下的VBench I2V任务中也取得了90.1的最佳结果。通过提供一个透明且可扩展的方案来构建前沿的视频生成模型，我们的目标是推动未来的研究，并加速迈向更灵活可靠的视频生成解决方案的步伐。
URL
https://arxiv.org/abs/2412.07730
PDF
-
Efficient Continuous Video Flow Model for Video Prediction
2024-12-07 12:11:25Gaurav Shrivastava, Abhinav ShrivastavaAbstract
Multi-step prediction models, such as diffusion and rectified flow models, have emerged as state-of-the-art solutions for generation tasks. However, these models exhibit higher latency in sampling new frames compared to single-step methods. This latency issue becomes a significant bottleneck when adapting such methods for video prediction tasks, given that a typical 60-second video comprises approximately 1.5K frames. In this paper, we propose a novel approach to modeling the multi-step process, aimed at alleviating latency constraints and facilitating the adaptation of such processes for video prediction tasks. Our approach not only reduces the number of sample steps required to predict the next frame but also minimizes computational demands by reducing the model size to one-third of the original size. We evaluate our method on standard video prediction datasets, including KTH, BAIR action robot, Human3.6M and UCF101, demonstrating its efficacy in achieving state-of-the-art performance on these benchmarks.
Abstract (translated)
多步预测模型，如扩散模型和修正流模型，已作为生成任务的最新解决方案出现。然而，与单步方法相比，这些模型在采样新帧时表现出更高的延迟。考虑到一个典型的60秒视频包含大约1500帧，在将此类方法适应于视频预测任务时，这种延迟问题成为一个显著的瓶颈。本文提出了一种新的多步过程建模方法，旨在缓解这一延迟限制并促进其向视频预测任务的适应。我们的方法不仅减少了预测下一帧所需的采样步骤数量，还通过将模型大小减少到原来的三分之一来降低计算需求。我们在标准的视频预测数据集上评估了该方法，包括KTH、BAIR动作机器人、Human3.6M和UCF101，结果表明它在这些基准测试中达到了最先进的性能水平。
URL
https://arxiv.org/abs/2412.05633
PDF
-
Continuous Video Process: Modeling Videos as Continuous Multi-Dimensional Processes for Video Prediction
2024-12-06 10:34:50Gaurav Shrivastava, Abhinav ShrivastavaAbstract
Diffusion models have made significant strides in image generation, mastering tasks such as unconditional image synthesis, text-image translation, and image-to-image conversions. However, their capability falls short in the realm of video prediction, mainly because they treat videos as a collection of independent images, relying on external constraints such as temporal attention mechanisms to enforce temporal coherence. In our paper, we introduce a novel model class, that treats video as a continuous multi-dimensional process rather than a series of discrete frames. We also report a reduction of 75\% sampling steps required to sample a new frame thus making our framework more efficient during the inference time. Through extensive experimentation, we establish state-of-the-art performance in video prediction, validated on benchmark datasets including KTH, BAIR, Human3.6M, and UCF101. Navigate to the project page this https URL for video results.}
Abstract (translated)
扩散模型在图像生成方面取得了显著进展，掌握了诸如无条件图像合成、文本-图像翻译和图像到图像转换等任务。然而，在视频预测领域，它们的能力有所不足，主要是因为这些模型将视频视为一系列独立的图像，并依赖外部约束（如时间注意力机制）来维持时间连贯性。在我们的论文中，我们引入了一种新型模型类别，该类模型将视频视为一个连续多维过程，而不是一系列离散帧。我们还报告称，在采样新帧所需的步骤上减少了75%，从而提高了推理阶段的效率。通过广泛的实验，我们在视频预测方面达到了业界领先水平，并在KTH、BAIR、Human3.6M和UCF101等基准数据集上进行了验证。访问此项目页面（[此处为链接]），可查看视频结果。
URL
https://arxiv.org/abs/2412.04929
PDF
-
Lightweight Stochastic Video Prediction via Hybrid Warping
2024-12-04 06:33:27Kazuki Kotoyori, Shota Hirose, Heming Sun, Jiro KattoAbstract
Accurate video prediction by deep neural networks, especially for dynamic regions, is a challenging task in computer vision for critical applications such as autonomous driving, remote working, and telemedicine. Due to inherent uncertainties, existing prediction models often struggle with the complexity of motion dynamics and occlusions. In this paper, we propose a novel stochastic long-term video prediction model that focuses on dynamic regions by employing a hybrid warping strategy. By integrating frames generated through forward and backward warpings, our approach effectively compensates for the weaknesses of each technique, improving the prediction accuracy and realism of moving regions in videos while also addressing uncertainty by making stochastic predictions that account for various motions. Furthermore, considering real-time predictions, we introduce a MobileNet-based lightweight architecture into our model. Our model, called SVPHW, achieves state-of-the-art performance on two benchmark datasets.
Abstract (translated)
URL
https://arxiv.org/abs/2412.03061
PDF
