Abstract
Multi-step prediction models, such as diffusion and rectified flow models, have emerged as state-of-the-art solutions for generation tasks. However, these models exhibit higher latency in sampling new frames compared to single-step methods. This latency issue becomes a significant bottleneck when adapting such methods for video prediction tasks, given that a typical 60-second video comprises approximately 1.5K frames. In this paper, we propose a novel approach to modeling the multi-step process, aimed at alleviating latency constraints and facilitating the adaptation of such processes for video prediction tasks. Our approach not only reduces the number of sample steps required to predict the next frame but also minimizes computational demands by reducing the model size to one-third of the original size. We evaluate our method on standard video prediction datasets, including KTH, BAIR action robot, Human3.6M and UCF101, demonstrating its efficacy in achieving state-of-the-art performance on these benchmarks.
Abstract (translated)
多步预测模型，如扩散模型和修正流模型，已作为生成任务的最新解决方案出现。然而，与单步方法相比，这些模型在采样新帧时表现出更高的延迟。考虑到一个典型的60秒视频包含大约1500帧，在将此类方法适应于视频预测任务时，这种延迟问题成为一个显著的瓶颈。本文提出了一种新的多步过程建模方法，旨在缓解这一延迟限制并促进其向视频预测任务的适应。我们的方法不仅减少了预测下一帧所需的采样步骤数量，还通过将模型大小减少到原来的三分之一来降低计算需求。我们在标准的视频预测数据集上评估了该方法，包括KTH、BAIR动作机器人、Human3.6M和UCF101，结果表明它在这些基准测试中达到了最先进的性能水平。
URL
https://arxiv.org/abs/2412.05633
