Abstract
Text-to-image generation has evolved beyond single monolithic models to complex multi-component pipelines. These combine fine-tuned generators, adapters, upscaling blocks and even editing steps, leading to significant improvements in image quality. However, their effective design requires substantial expertise. Recent approaches have shown promise in automating this process through large language models (LLMs), but they suffer from two critical limitations: extensive computational requirements from generating images with hundreds of predefined pipelines, and poor generalization beyond memorized training examples. We introduce a novel reinforcement learning-based framework that addresses these inefficiencies. Our approach first trains an ensemble of reward models capable of predicting image quality scores directly from prompt-workflow combinations, eliminating the need for costly image generation during training. We then implement a two-phase training strategy: initial workflow vocabulary training followed by GRPO-based optimization that guides the model toward higher-performing regions of the workflow space. Additionally, we incorporate a classifier-free guidance based enhancement technique that extrapolates along the path between the initial and GRPO-tuned models, further improving output quality. We validate our approach through a set of comparisons, showing that it can successfully create new flows with greater diversity and lead to superior image quality compared to existing baselines.
Abstract (translated)
文本到图像的生成技术已经从单一的整体模型发展到了复杂的多组件流水线。这些流水线结合了经过微调的生成器、适配器、上采样模块以及编辑步骤，从而显著提升了图像质量。然而，这种流水线的有效设计需要大量的专业知识。最近的研究表明，通过大型语言模型（LLMs）可以实现自动化这一过程，但是这种方法存在两个关键限制：一是由于需要生成数百个预定义的流水线来完成任务而产生的巨大计算需求；二是对超出训练样本的记忆泛化能力较差。我们提出了一种基于强化学习的新框架，以解决这些效率问题。 我们的方法首先训练一组奖励模型，这些模型能够直接从提示-工作流组合中预测图像质量得分，从而在训练过程中消除了昂贵的图像生成需求。接下来，我们实施了两阶段的训练策略：首先是初始的工作流程词汇表训练，然后是基于GRPO（Guided Reinforcement Policy Optimization）的优化，该优化引导模型向表现更好的工作流程空间区域发展。此外，我们还引入了一种无需分类器指导的技术改进方法，通过在初始模型与经过GRPO调整后的模型之间的路径上进行外推来进一步提升输出质量。 为了验证我们的方法的有效性，我们通过一系列对比实验进行了测试，并展示了该方法能够成功创建多样化的新型工作流程，并且相比现有基准，在图像质量方面表现出色。
URL
https://arxiv.org/abs/2505.21478
