Abstract
Diffusion models have made significant strides in image generation, mastering tasks such as unconditional image synthesis, text-image translation, and image-to-image conversions. However, their capability falls short in the realm of video prediction, mainly because they treat videos as a collection of independent images, relying on external constraints such as temporal attention mechanisms to enforce temporal coherence. In our paper, we introduce a novel model class, that treats video as a continuous multi-dimensional process rather than a series of discrete frames. We also report a reduction of 75\% sampling steps required to sample a new frame thus making our framework more efficient during the inference time. Through extensive experimentation, we establish state-of-the-art performance in video prediction, validated on benchmark datasets including KTH, BAIR, Human3.6M, and UCF101. Navigate to the project page this https URL for video results.}
Abstract (translated)
扩散模型在图像生成方面取得了显著进展，掌握了诸如无条件图像合成、文本-图像翻译和图像到图像转换等任务。然而，在视频预测领域，它们的能力有所不足，主要是因为这些模型将视频视为一系列独立的图像，并依赖外部约束（如时间注意力机制）来维持时间连贯性。在我们的论文中，我们引入了一种新型模型类别，该类模型将视频视为一个连续多维过程，而不是一系列离散帧。我们还报告称，在采样新帧所需的步骤上减少了75%，从而提高了推理阶段的效率。通过广泛的实验，我们在视频预测方面达到了业界领先水平，并在KTH、BAIR、Human3.6M和UCF101等基准数据集上进行了验证。访问此项目页面（[此处为链接]），可查看视频结果。
URL
https://arxiv.org/abs/2412.04929
