Abstract
In this paper, we propose a unified layout planning and image generation model, PlanGen, which can pre-plan spatial layout conditions before generating images. Unlike previous diffusion-based models that treat layout planning and layout-to-image as two separate models, PlanGen jointly models the two tasks into one autoregressive transformer using only next-token prediction. PlanGen integrates layout conditions into the model as context without requiring specialized encoding of local captions and bounding box coordinates, which provides significant advantages over the previous embed-and-pool operations on layout conditions, particularly when dealing with complex layouts. Unified prompting allows PlanGen to perform multitasking training related to layout, including layout planning, layout-to-image generation, image layout understanding, etc. In addition, PlanGen can be seamlessly expanded to layout-guided image manipulation thanks to the well-designed modeling, with teacher-forcing content manipulation policy and negative layout guidance. Extensive experiments verify the effectiveness of our PlanGen in multiple layoutrelated tasks, showing its great potential. Code is available at: this https URL.
Abstract (translated)
在这篇论文中，我们提出了一种统一的布局规划和图像生成模型——PlanGen。该模型能够在生成图像之前预先计划空间布局条件。与以往基于扩散的方法将布局规划和从布局到图像转换视为两个独立模型不同，PlanGen 将这两个任务整合为一个自回归变压器模型，并仅通过下一个标记预测来实现这一目标。PlanGen 在不专门编码局部说明文本和边界框坐标的情况下，将布局条件集成到模型上下文中，这在处理复杂布局时相比之前的嵌入并池化操作提供了显著的优势。 统一的提示机制使 PlanGen 能够进行与布局相关的多任务训练，包括布局规划、从布局生成图像、理解图像中的布局等。此外，由于其设计得当，PlanGen 还可以无缝扩展到基于布局引导的图像编辑功能，并且通过教师强制内容操作策略和负向布局指导来实现这一点。 广泛的实验验证了 PlanGen 在多个与布局相关的任务中的有效性，展示了其巨大的潜力。代码可在以下网址获取：[此处插入具体链接]。
URL
https://arxiv.org/abs/2503.10127
