Abstract
The field of video generation has made remarkable advancements, yet there remains a pressing need for a clear, systematic recipe that can guide the development of robust and scalable models. In this work, we present a comprehensive study that systematically explores the interplay of model architectures, training recipes, and data curation strategies, culminating in a simple and scalable text-image-conditioned video generation method, named STIV. Our framework integrates image condition into a Diffusion Transformer (DiT) through frame replacement, while incorporating text conditioning via a joint image-text conditional classifier-free guidance. This design enables STIV to perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks simultaneously. Additionally, STIV can be easily extended to various applications, such as video prediction, frame interpolation, multi-view generation, and long video generation, etc. With comprehensive ablation studies on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V, surpassing both leading open and closed-source models like CogVideoX-5B, Pika, Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and extensible recipe for building cutting-edge video generation models, we aim to empower future research and accelerate progress toward more versatile and reliable video generation solutions.
Abstract (translated)
视频生成领域取得了显著的进步，但仍迫切需要一个清晰、系统的方案来指导构建强大且可扩展的模型。在这项工作中，我们进行了一项全面的研究，系统地探讨了模型架构、训练方法和数据整理策略之间的相互作用，最终形成了一种简单而可扩展的文本-图像条件视频生成方法，名为STIV。我们的框架通过帧替换将图像条件整合到扩散转换器（DiT）中，并通过联合的图像-文本条件无引导分类器引入文本条件。这种设计使STIV能够同时执行从文本到视频（T2V）和从文本-图像到视频（TI2V）的任务。此外，STIV可以轻松扩展到各种应用，例如视频预测、帧插值、多视图生成和长视频生成等。通过对T2I、T2V和TI2V进行详尽的消融研究，尽管设计简单，STIV仍表现出色。一个87亿参数模型在512分辨率下，在VBench T2V任务上达到了83.1分，超过了领先的开源和闭源模型如CogVideoX-5B、Pika、Kling和Gen-3等。相同规模的模型在512分辨率下的VBench I2V任务中也取得了90.1的最佳结果。通过提供一个透明且可扩展的方案来构建前沿的视频生成模型，我们的目标是推动未来的研究，并加速迈向更灵活可靠的视频生成解决方案的步伐。
URL
https://arxiv.org/abs/2412.07730
