Abstract
We train models to Predict Ego-centric Video from human Actions (PEVA), given the past video and an action represented by the relative 3D body pose. By conditioning on kinematic pose trajectories, structured by the joint hierarchy of the body, our model learns to simulate how physical human actions shape the environment from a first-person point of view. We train an auto-regressive conditional diffusion transformer on Nymeria, a large-scale dataset of real-world egocentric video and body pose capture. We further design a hierarchical evaluation protocol with increasingly challenging tasks, enabling a comprehensive analysis of the model's embodied prediction and control abilities. Our work represents an initial attempt to tackle the challenges of modeling complex real-world environments and embodied agent behaviors with video prediction from the perspective of a human.
Abstract (translated)
我们训练了一种模型，用于从人类动作预测第一人称视角的视频（PEVA），该模型利用过去视频和由相对3D人体姿态表示的动作来进行工作。通过基于人体关节层次结构约束的动力学姿态轨迹进行条件设定，我们的模型学会了模拟物理人类行为如何以第一人称视角塑造环境。我们在Nymeria上训练了一种自回归条件扩散变换器，Nymeria是一个大规模的真实世界第一人称视频和身体姿态捕捉数据集。我们进一步设计了一个分层评估协议，该协议包含逐渐增加难度的任务，从而能够对模型的具身预测与控制能力进行全面分析。我们的工作代表了首次尝试从人类视角出发通过视频预测来应对模拟复杂现实环境及具身智能体行为挑战的努力。
URL
https://arxiv.org/abs/2506.21552
