Abstract
In egocentric scenarios, anticipating both the next action and its visual outcome is essential for understanding human-object interactions and for enabling robotic planning. However, existing paradigms fall short of jointly modeling these aspects. Vision-Language-Action (VLA) models focus on action prediction but lack explicit modeling of how actions influence the visual scene, while video prediction models generate future frames without conditioning on specific actions, often resulting in implausible or contextually inconsistent outcomes. To bridge this gap, we propose a unified two-stage predictive framework that jointly models action and visual future in egocentric scenarios, conditioned on hand trajectories. In the first stage, we perform consecutive state modeling to process heterogeneous inputs (visual observations, language, and action history) and explicitly predict future hand trajectories. In the second stage, we introduce causal cross-attention to fuse multi-modal cues, leveraging inferred action signals to guide an image-based Latent Diffusion Model (LDM) for frame-by-frame future video generation. Our approach is the first unified model designed to handle both egocentric human activity understanding and robotic manipulation tasks, providing explicit predictions of both upcoming actions and their visual consequences. Extensive experiments on Ego4D, BridgeData, and RLBench demonstrate that our method outperforms state-of-the-art baselines in both action prediction and future video synthesis.
Abstract (translated)
在以自我为中心的情境中，预测下一个动作及其视觉结果对于理解人与物体的互动以及使机器人计划成为可能至关重要。然而，现有的范式无法同时建模这些方面。Vision-Language-Action（VLA）模型专注于动作预测，但未能明确地建模动作如何影响视觉场景；而视频预测模型在不基于特定动作的情况下生成未来帧，往往导致结果不合理或与上下文不符。为了弥合这一差距，我们提出了一种统一的两阶段预测框架，在以自我为中心的情境中联合建模动作和视觉未来的模式，并依据手部轨迹进行条件设置。在第一阶段，我们执行连续状态建模来处理异构输入（视觉观察、语言和动作历史），并明确地预测未来的手部轨迹。在第二阶段，我们引入因果交叉注意力机制融合多模态线索，利用推断出的动作信号指导基于图像的潜在扩散模型进行逐帧未来的视频生成。我们的方法是第一个统一模型，旨在同时处理以自我为中心的人类活动理解和机器人操作任务，并明确地预测即将发生的动作及其视觉后果。在Ego4D、BridgeData和RLBench数据集上的广泛实验表明，我们提出的方法在动作预测和未来视频合成方面都优于最先进的基线方法。
URL
https://arxiv.org/abs/2508.19852
