Image_Generation
-
Vec2Face+ for Face Dataset Generation
2025-07-23 04:34:56Haiyu Wu, Jaskirat Singh, Sicong Tian, Liang Zheng, Kevin W. BowyerAbstract
When synthesizing identities as face recognition training data, it is generally believed that large inter-class separability and intra-class attribute variation are essential for synthesizing a quality dataset. % This belief is generally correct, and this is what we aim for. However, when increasing intra-class variation, existing methods overlook the necessity of maintaining intra-class identity consistency. % To address this and generate high-quality face training data, we propose Vec2Face+, a generative model that creates images directly from image features and allows for continuous and easy control of face identities and attributes. Using Vec2Face+, we obtain datasets with proper inter-class separability and intra-class variation and identity consistency using three strategies: 1) we sample vectors sufficiently different from others to generate well-separated identities; 2) we propose an AttrOP algorithm for increasing general attribute variations; 3) we propose LoRA-based pose control for generating images with profile head poses, which is more efficient and identity-preserving than AttrOP. % Our system generates VFace10K, a synthetic face dataset with 10K identities, which allows an FR model to achieve state-of-the-art accuracy on seven real-world test sets. Scaling the size to 4M and 12M images, the corresponding VFace100K and VFace300K datasets yield higher accuracy than the real-world training dataset, CASIA-WebFace, on five real-world test sets. This is the first time a synthetic dataset beats the CASIA-WebFace in average accuracy. In addition, we find that only 1 out of 11 synthetic datasets outperforms random guessing (\emph{i.e., 50\%}) in twin verification and that models trained with synthetic identities are more biased than those trained with real identities. Both are important aspects for future investigation.
Abstract (translated)
在将身份合成用作面部识别训练数据时，通常认为大类间分离性和类内属性变化对于生成高质量的数据集至关重要。这种看法通常是正确的，这也是我们的目标所在。然而，在增加类内变化的同时，现有方法往往忽略了保持类内身份一致性的重要性。为了应对这一挑战并生成高质量的面部训练数据，我们提出了一种名为Vec2Face+的生成模型，该模型可以直接从图像特征创建图像，并允许对人脸身份和属性进行连续且易于控制的操作。通过使用Vec2Face+，我们采用三种策略获得了具备适当类间分离性和类内变化及一致性身份的数据集：1）采样足够不同于其他样本的向量以生成具有较好隔离性的身份；2）提出了一种AttrOP算法来增加一般属性的变化；3）提出了基于LoRA的姿态控制方法，用于生成带有侧脸姿态的图像，这种方法比AttrOP更高效且更能保持身份一致性。 我们的系统产生了VFace10K，这是一个拥有10,000个不同身份的人工合成面部数据集，在七个真实世界的测试集中使面部识别模型达到了最先进的精度水平。当我们扩展该数据集规模至4M和12M图像时，对应的VFace100K和VFace300K数据集在五个实际测试集合上比现实世界训练数据集CASIA-WebFace的平均准确度更高。这是首次合成数据集在平均精度上超越了CASIA-WebFace。 此外，我们发现只有11个合成数据集中的一项超过了随机猜测（即50%）的表现，在双胞胎验证任务中。另外，使用合成身份训练出的模型比使用真实身份训练出的模型更加偏颇。这两者都是未来研究的重要方面。
URL
https://arxiv.org/abs/2507.17192
PDF
-
Bringing Balance to Hand Shape Classification: Mitigating Data Imbalance Through Generative Models
2025-07-22 20:41:29Gaston Gustavo Rios, Pedro Dal Bianco, Franco Ronchetti, Facundo Quiroga, Oscar Stanchi, Santiago Ponte Ah\'on, Waldo Hasperu\'eAbstract
Most sign language handshape datasets are severely limited and unbalanced, posing significant challenges to effective model training. In this paper, we explore the effectiveness of augmenting the training data of a handshape classifier by generating synthetic data. We use an EfficientNet classifier trained on the RWTH German sign language handshape dataset, which is small and heavily unbalanced, applying different strategies to combine generated and real images. We compare two Generative Adversarial Networks (GAN) architectures for data generation: ReACGAN, which uses label information to condition the data generation process through an auxiliary classifier, and SPADE, which utilizes spatially-adaptive normalization to condition the generation on pose information. ReACGAN allows for the generation of realistic images that align with specific handshape labels, while SPADE focuses on generating images with accurate spatial handshape configurations. Our proposed techniques improve the current state-of-the-art accuracy on the RWTH dataset by 5%, addressing the limitations of small and unbalanced datasets. Additionally, our method demonstrates the capability to generalize across different sign language datasets by leveraging pose-based generation trained on the extensive HaGRID dataset. We achieve comparable performance to single-source trained classifiers without the need for retraining the generator.
Abstract (translated)
大多数手语手势数据集在规模和平衡性上都存在严重限制，这给有效的模型训练带来了重大挑战。本文探讨了通过生成合成数据来增强手势分类器训练数据有效性的方法。我们使用了一个基于RWTH德国手语手势数据集（该数据集较小且不平衡）训练的EfficientNet分类器，并应用不同的策略将生成的数据与真实图像结合。我们比较了两种用于数据生成的生成对抗网络（GAN）架构：ReACGAN，通过辅助分类器利用标签信息来调节数据生成过程；以及SPADE，它采用空间自适应归一化来根据姿势信息调整生成过程。ReACGAN能够生成与特定手势标签相符的真实图像，而SPADE则专注于生成具有准确手部姿态配置的图像。 我们提出的方法在RWTH数据集上将当前最先进的准确性提高了5%，解决了小规模和不平衡数据集的问题。此外，我们的方法还展示了跨不同手语数据集泛化的能力，通过基于姿势的生成技术，在庞大的HaGRID数据集上进行训练后实现这一点。我们在无需重新训练生成器的情况下，达到了与单一来源训练分类器相当的表现。 综上所述，本文提出的方法为解决手势识别中数据集限制问题提供了一种有效途径，并展示了其在不同手语数据集中泛化的潜力。
URL
https://arxiv.org/abs/2507.17008
PDF
-
Resurrect Mask AutoRegressive Modeling for Efficient and Scalable Image Generation
2025-07-17 12:02:38Yi Xin, Le Zhuo, Qi Qin, Siqi Luo, Yuewen Cao, Bin Fu, Yangfan He, Hongsheng Li, Guangtao Zhai, Xiaohong Liu, Peng GaoAbstract
AutoRegressive (AR) models have made notable progress in image generation, with Masked AutoRegressive (MAR) models gaining attention for their efficient parallel decoding. However, MAR models have traditionally underperformed when compared to standard AR models. This study refines the MAR architecture to improve image generation quality. We begin by evaluating various image tokenizers to identify the most effective one. Subsequently, we introduce an improved Bidirectional LLaMA architecture by replacing causal attention with bidirectional attention and incorporating 2D RoPE, which together form our advanced model, MaskGIL. Scaled from 111M to 1.4B parameters, MaskGIL achieves a FID score of 3.71, matching state-of-the-art AR models in the ImageNet 256x256 benchmark, while requiring only 8 inference steps compared to the 256 steps of AR models. Furthermore, we develop a text-driven MaskGIL model with 775M parameters for generating images from text at various resolutions. Beyond image generation, MaskGIL extends to accelerate AR-based generation and enable real-time speech-to-image conversion. Our codes and models are available at this https URL.
Abstract (translated)
自回归（AutoRegressive，AR）模型在图像生成领域取得了显著进展，而遮蔽自回归（Masked AutoRegressive，MAR）模型由于其高效的并行解码能力受到了广泛关注。然而，与标准的AR模型相比，传统的MAR模型性能较差。本研究对MAR架构进行了改进，以提高图像生成质量。我们首先评估了各种图像标记器，以确定最有效的标记器。随后，我们通过将因果注意力替换为双向注意力，并引入2D RoPE来改进Bidirectional LLaMA架构，从而形成我们的先进模型MaskGIL。从1.11亿参数扩展到14亿参数规模的MaskGIL，在ImageNet 256x256基准测试中达到了3.71的FID评分，与最先进的AR模型相匹配，同时仅需8个推理步骤，而相比之下AR模型需要256个步骤。 此外，我们开发了一个具有7.75亿参数、基于文本驱动的MaskGIL模型，用于在各种分辨率下从文本生成图像。除了图像生成之外，MaskGIL还扩展了基于自回归的生成加速，并实现了实时语音转图像转换。我们的代码和模型可在以下网址获取：[此处插入链接]（请将实际URL填入）。
URL
https://arxiv.org/abs/2507.13032
PDF
-
ID-EA: Identity-driven Text Enhancement and Adaptation with Textual Inversion for Personalized Text-to-Image Generation
2025-07-16 07:42:02Hyun-Jun Jin, Young-Eun Kim, Seong-Whan LeeAbstract
Recently, personalized portrait generation with a text-to-image diffusion model has significantly advanced with Textual Inversion, emerging as a promising approach for creating high-fidelity personalized images. Despite its potential, current Textual Inversion methods struggle to maintain consistent facial identity due to semantic misalignments between textual and visual embedding spaces regarding identity. We introduce ID-EA, a novel framework that guides text embeddings to align with visual identity embeddings, thereby improving identity preservation in a personalized generation. ID-EA comprises two key components: the ID-driven Enhancer (ID-Enhancer) and the ID-conditioned Adapter (ID-Adapter). First, the ID-Enhancer integrates identity embeddings with a textual ID anchor, refining visual identity embeddings derived from a face recognition model using representative text embeddings. Then, the ID-Adapter leverages the identity-enhanced embedding to adapt the text condition, ensuring identity preservation by adjusting the cross-attention module in the pre-trained UNet model. This process encourages the text features to find the most related visual clues across the foreground snippets. Extensive quantitative and qualitative evaluations demonstrate that ID-EA substantially outperforms state-of-the-art methods in identity preservation metrics while achieving remarkable computational efficiency, generating personalized portraits approximately 15 times faster than existing approaches.
Abstract (translated)
最近，使用文本到图像扩散模型进行个性化肖像生成在引入Textual Inversion技术后取得了显著进展，成为创建高保真个性化图像的一种有前景的方法。尽管具有巨大潜力，现有的Textual Inversion方法却难以保持面部身份的一致性，因为文本和视觉嵌入空间之间关于身份的语义不匹配导致了这个问题。我们提出了ID-EA（Identity-Driven Embedding Alignment），这是一种新的框架，它引导文本嵌入与视觉身份嵌入对齐，从而在个性化生成中提高身份保存的效果。 ID-EA包括两个关键组成部分：由身份驱动的增强器（ID-Enhancer）和根据身份条件适配器（ID-Adapter）。首先，ID-Enhancer 将身份嵌入与文本 ID 锚点集成起来，并使用来自面部识别模型的视觉身份嵌入对代表性文本嵌入进行细化。然后，ID-Adapter 利用增强后的身份嵌入来调整文本条件，在预训练的 UNet 模型中通过调节交叉注意力模块确保身份保存。这一过程鼓励文本特征找到前景片段中的最相关视觉线索。 广泛的定量和定性评估表明，与现有方法相比，ID-EA 在身份保持度量方面显著超越了最先进的技术，并且实现了令人瞩目的计算效率，在生成个性化肖像时比现有的方法快大约15倍。
URL
https://arxiv.org/abs/2507.11990
PDF
-
RaDL: Relation-aware Disentangled Learning for Multi-Instance Text-to-Image Generation
2025-07-16 06:28:20Geon Park, Seon Bin Kim, Gunho Jung, Seong-Whan LeeAbstract
With recent advancements in text-to-image (T2I) models, effectively generating multiple instances within a single image prompt has become a crucial challenge. Existing methods, while successful in generating positions of individual instances, often struggle to account for relationship discrepancy and multiple attributes leakage. To address these limitations, this paper proposes the relation-aware disentangled learning (RaDL) framework. RaDL enhances instance-specific attributes through learnable parameters and generates relation-aware image features via Relation Attention, utilizing action verbs extracted from the global prompt. Through extensive evaluations on benchmarks such as COCO-Position, COCO-MIG, and DrawBench, we demonstrate that RaDL outperforms existing methods, showing significant improvements in positional accuracy, multiple attributes consideration, and the relationships between instances. Our results present RaDL as the solution for generating images that consider both the relationships and multiple attributes of each instance within the multi-instance image.
Abstract (translated)
随着文本到图像（T2I）模型的近期进展，有效地在一个图片提示中生成多个实例已经成为一个重要的挑战。现有的方法虽然在生成单个实例的位置方面取得了成功，但在处理关系差异和多重属性泄露时往往力不从心。为了应对这些限制，本文提出了一种基于关系感知解耦学习（RaDL）框架。通过可学习的参数，RaDL增强了每个实例特有的属性，并通过使用从全局提示中提取的动作动词来生成具有关系意识的图像特征。利用Relation Attention机制，该方法能够更好地理解和处理实例之间的关系以及多重属性。 在COCO-Position、COCO-MIG和DrawBench等基准测试上的广泛评估表明，RaDL优于现有的方法，在位置准确性、考虑多重属性及实例间的关系方面均有显著改进。我们的结果将RaDL视为生成同时考虑每个实例之间关系及其多重属性的多实例图像的最佳解决方案。
URL
https://arxiv.org/abs/2507.11947
PDF
-
Towards Evaluating Robustness of Prompt Adherence in Text to Image Models
2025-07-09 18:40:17Sujith Vemishetty, Advitiya Arora, Anupama SharmaAbstract
The advancements in the domain of LLMs in recent years have surprised many, showcasing their remarkable capabilities and diverse applications. Their potential applications in various real-world scenarios have led to significant research on their reliability and effectiveness. On the other hand, multimodal LLMs and Text-to-Image models have only recently gained prominence, especially when compared to text-only LLMs. Their reliability remains constrained due to insufficient research on assessing their performance and robustness. This paper aims to establish a comprehensive evaluation framework for Text-to-Image models, concentrating particularly on their adherence to prompts. We created a novel dataset that aimed to assess the robustness of these models in generating images that conform to the specified factors of variation in the input text prompts. Our evaluation studies present findings on three variants of Stable Diffusion models: Stable Diffusion 3 Medium, Stable Diffusion 3.5 Large, and Stable Diffusion 3.5 Large Turbo, and two variants of Janus models: Janus Pro 1B and Janus Pro 7B. We introduce a pipeline that leverages text descriptions generated by the gpt-4o model for our ground-truth images, which are then used to generate artificial images by passing these descriptions to the Text-to-Image models. We then pass these generated images again through gpt-4o using the same system prompt and compare the variation between the two descriptions. Our results reveal that these models struggle to create simple binary images with only two factors of variation: a simple geometric shape and its location. We also show, using pre-trained VAEs on our dataset, that they fail to generate images that follow our input dataset distribution.
Abstract (translated)
近年来，大型语言模型（LLM）领域取得了许多令人惊讶的进步，展示了它们在多种场景中的显著能力和广泛应用。由于这些模型在实际应用场景中显示出巨大的潜力，因此对其可靠性和有效性进行了大量的研究。相比之下，多模态的LLM和文本到图像模型直到最近才开始受到关注，特别是在与纯文本LLM相比时。尽管如此，由于缺乏对它们性能和鲁棒性的评估研究，其可靠性仍然受限。 本文旨在为文本到图像（Text-to-Image）模型建立一个全面的评价框架，特别关注这些模型在生成符合输入文本提示中指定因素变化的图像方面的表现能力。为此，我们创建了一个新的数据集，用于评估这些模型在上述条件下的稳健性。我们的研究对三种Stable Diffusion模型进行了评测：Stable Diffusion 3 Medium、Stable Diffusion 3.5 Large以及Stable Diffusion 3.5 Large Turbo；同时还有两种Janus模型：Janus Pro 1B和Janus Pro 7B。 为了进行这项评估，我们引入了一个管道系统，该系统利用gpt-4o生成的文本描述作为真实图像的基础，并将这些描述传递给Text-to-Image模型以生成人工图像。然后，我们将生成的图像再次通过相同的系统提示传递给gpt-4o，并比较两个描述之间的变化。 我们的结果表明，在只有两种因素（一个简单的几何形状及其位置）发生二元变化的情况下，这些模型在创建简单二元图象方面遇到困难。此外，我们还使用预训练的变分自编码器（VAE），并在我们的数据集上展示了它们无法生成符合输入数据分布特性的图像。 总的来说，这项研究揭示了现有Text-to-Image模型的一些局限性，并为进一步改善这些模型提供了有价值的见解和建议。
URL
https://arxiv.org/abs/2507.08039
PDF
-
Concept Pinpoint Eraser for Text-to-image Diffusion Models via Residual Attention Gate
2025-06-28 08:17:19Byung Hyun Lee, Sungjin Lim, Seunggyu Lee, Dong Un Kang, Se Young ChunAbstract
Remarkable progress in text-to-image diffusion models has brought a major concern about potentially generating images on inappropriate or trademarked concepts. Concept erasing has been investigated with the goals of deleting target concepts in diffusion models while preserving other concepts with minimal distortion. To achieve these goals, recent concept erasing methods usually fine-tune the cross-attention layers of diffusion models. In this work, we first show that merely updating the cross-attention layers in diffusion models, which is mathematically equivalent to adding \emph{linear} modules to weights, may not be able to preserve diverse remaining concepts. Then, we propose a novel framework, dubbed Concept Pinpoint Eraser (CPE), by adding \emph{nonlinear} Residual Attention Gates (ResAGs) that selectively erase (or cut) target concepts while safeguarding remaining concepts from broad distributions by employing an attention anchoring loss to prevent the forgetting. Moreover, we adversarially train CPE with ResAG and learnable text embeddings in an iterative manner to maximize erasing performance and enhance robustness against adversarial attacks. Extensive experiments on the erasure of celebrities, artistic styles, and explicit contents demonstrated that the proposed CPE outperforms prior arts by keeping diverse remaining concepts while deleting the target concepts with robustness against attack prompts. Code is available at this https URL
Abstract (translated)
文本到图像的扩散模型取得了显著的进步，但也引发了人们对生成不适当或受商标保护的概念图片的担忧。概念擦除技术的研究目标是在删除特定概念的同时尽可能地保留其他概念而不造成重大失真。最近的方法通常通过微调扩散模型中的交叉注意力层来实现这一目的。 然而，在这项工作中，研究者们首先展示了仅仅更新扩散模型中的交叉注意力层（这在数学上等同于向权重中添加线性模块）可能不足以保护多样化存在的其他概念。为了解决这个问题，他们提出了一种新的框架——概念定点擦除器（Concept Pinpoint Eraser, CPE）。该框架通过添加非线性的残差注意门（Residual Attention Gates, ResAGs），选择性地删除目标概念，同时利用注意力锚定损失来防止遗忘，从而保护其他广泛分布的概念。此外，研究者们还采用对抗训练方法迭代式地优化CPE及其可学习的文本嵌入，以最大化擦除性能并增强对对抗攻击的鲁棒性。 通过在名人、艺术风格和显性内容上的大量实验验证了所提出的CPE框架，在删除目标概念的同时能够保持多样化存在的其他概念，并且具有较强的抗干扰能力。代码可在提供的链接处获取。
URL
https://arxiv.org/abs/2506.22806
PDF
-
Reproducible Evaluation of Camera Auto-Exposure Methods in the Field: Platform, Benchmark and Lessons Learned
2025-06-19 14:01:01Olivier Gamache, Jean-Michel Fortin, Mat\v{e}j Boxan, Fran\c{c}ois Pomerleau, Philippe Gigu\`ereAbstract
Standard datasets often present limitations, particularly due to the fixed nature of input data sensors, which makes it difficult to compare methods that actively adjust sensor parameters to suit environmental conditions. This is the case with Automatic-Exposure (AE) methods, which rely on environmental factors to influence the image acquisition process. As a result, AE methods have traditionally been benchmarked in an online manner, rendering experiments non-reproducible. Building on our prior work, we propose a methodology that utilizes an emulator capable of generating images at any exposure time. This approach leverages BorealHDR, a unique multi-exposure stereo dataset, along with its new extension, in which data was acquired along a repeated trajectory at different times of the day to assess the impact of changing illumination. In total, BorealHDR covers 13.4 km over 59 trajectories in challenging lighting conditions. The dataset also includes lidar-inertial-odometry-based maps with pose estimation for each image frame, as well as Global Navigation Satellite System (GNSS) data for comparison. We demonstrate that by using images acquired at various exposure times, we can emulate realistic images with a Root-Mean-Square Error (RMSE) below 1.78% compared to ground truth images. Using this offline approach, we benchmarked eight AE methods, concluding that the classical AE method remains the field's best performer. To further support reproducibility, we provide in-depth details on the development of our backpack acquisition platform, including hardware, electrical components, and performance specifications. Additionally, we share valuable lessons learned from deploying the backpack over more than 25 km across various environments. Our code and dataset are available online at this link: this https URL BorealHDR
Abstract (translated)
标准数据集通常存在一些限制，特别是由于输入数据传感器的固有特性难以改变，这使得比较那些能够根据环境条件调整传感器参数的方法变得困难。这种情况尤其体现在自动曝光（AE）方法中，这些方法依赖于环境因素来影响图像获取过程。因此，传统的AE方法在线进行基准测试，导致实验不可重复。 基于我们之前的工作，我们提出了一种利用模拟器生成任意曝光时间图像的方法。这种方法使用了BorealHDR数据集，这是一个独特的多曝光立体视觉数据集，以及其新的扩展版本，在这个版本中，数据是在一天中的不同时间段沿着相同的轨迹收集的，以评估光照变化的影响。总体而言，BorealHDR涵盖了59条轨迹、13.4公里的距离，并在挑战性的照明条件下进行了采集。该数据集还包括基于激光雷达-惯性里程计的地图以及每帧图像的姿态估计，还提供了全球导航卫星系统（GNSS）数据进行比较。 我们展示了通过使用不同曝光时间获取的图像，可以生成与真实情况相比误差低于1.78%的逼真图像。利用这种方法，我们在离线状态下对八种AE方法进行了基准测试，并得出结论：经典的AE方法仍然是表现最佳的方法。为了进一步支持可重复性研究，我们提供了有关背包采集平台开发的详细信息，包括硬件、电气组件和性能规范。此外，我们也分享了在超过25公里的各种环境下部署背包所获得的经验教训。 我们的代码和数据集可通过以下链接获取：[BorealHDR](https://this https URL)
URL
https://arxiv.org/abs/2506.18844
PDF
-
Disentangling 3D from Large Vision-Language Models for Controlled Portrait Generation
2025-06-16 21:26:45Nick Yiwen Huang, Akin Caliskan, Berkay Kicanaoglu, James Tompkin, Hyeongwoo KimAbstract
We consider the problem of disentangling 3D from large vision-language models, which we show on generative 3D portraits. This allows free-form text control of appearance attributes like age, hair style, and glasses, and 3D geometry control of face expression and camera pose. In this setting, we assume we use a pre-trained large vision-language model (LVLM; CLIP) to generate from a smaller 2D dataset with no additional paired labels and with a pre-defined 3D morphable model (FLAME). First, we disentangle using canonicalization to a 2D reference frame from a deformable neural 3D triplane representation. But another form of entanglement arises from the significant noise in the LVLM's embedding space that describes irrelevant features. This damages output quality and diversity, but we overcome this with a Jacobian regularization that can be computed efficiently with a stochastic approximator. Compared to existing methods, our approach produces portraits with added text and 3D control, where portraits remain consistent when either control is changed. Broadly, this approach lets creators control 3D generators on their own 2D face data without needing resources to label large data or train large models.
Abstract (translated)
我们考虑从大型视觉-语言模型中分离出三维信息的问题，并通过生成的三维肖像展示了这一过程。这使得我们可以用自由形式的文字控制外观属性（如年龄、发型和眼镜），并通过3D几何学控制面部表情和相机姿态。在这个设置下，假设我们使用一个预训练的大规模视觉-语言模型（LVLM；CLIP）从一个小的2D数据集中生成结果，并且该数据集没有额外配对标签，同时定义了一个预设的三维可变形模型（FLAME）。首先，我们通过将神经3D三平面表示规范到二维参考帧来实现分离。然而，另一种纠缠形式来自于LVLM嵌入空间中的大量噪声，这些噪声描述了无关特征。这种噪声会损害输出质量和多样性，但我们可以通过计算效率高的随机近似器进行雅可比正则化的方法克服这一问题。 与现有方法相比，我们的方法可以在生成的肖像中添加文本和3D控制，当改变任一控制时，肖像仍然保持一致性。总体而言，这种方法让创作者能够在其自己的2D面部数据上控制三维生成模型，并且无需为标注大量数据或训练大型模型投入资源。
URL
https://arxiv.org/abs/2506.14015
PDF
-
High-resolution efficient image generation from WiFi CSI using a pretrained latent diffusion model
2025-06-12 11:47:23Eshan Ramesh, Nishio TakayukiAbstract
We present LatentCSI, a novel method for generating images of the physical environment from WiFi CSI measurements that leverages a pretrained latent diffusion model (LDM). Unlike prior approaches that rely on complex and computationally intensive techniques such as GANs, our method employs a lightweight neural network to map CSI amplitudes directly into the latent space of an LDM. We then apply the LDM's denoising diffusion model to the latent representation with text-based guidance before decoding using the LDM's pretrained decoder to obtain a high-resolution image. This design bypasses the challenges of pixel-space image generation and avoids the explicit image encoding stage typically required in conventional image-to-image pipelines, enabling efficient and high-quality image synthesis. We validate our approach on two datasets: a wide-band CSI dataset we collected with off-the-shelf WiFi devices and cameras; and a subset of the publicly available MM-Fi dataset. The results demonstrate that LatentCSI outperforms baselines of comparable complexity trained directly on ground-truth images in both computational efficiency and perceptual quality, while additionally providing practical advantages through its unique capacity for text-guided controllability.
Abstract (translated)
我们提出了LatentCSI，这是一种利用预训练的潜在扩散模型（LDM）从WiFi CSI测量生成物理环境图像的新方法。与以往依赖复杂且计算密集的技术如GAN的方法不同，我们的方法采用轻量级神经网络直接将CSI幅度映射到LDM的潜在空间中。接着，我们使用带有文本引导的LDM去噪扩散模型对潜在表示进行处理，并通过预训练解码器解码以获得高分辨率图像。这种设计绕过了像素空间图像生成的挑战，并避免了传统图像到图像工作流通常需要的显式图像编码阶段，从而实现了高效且高质量的图像合成。 我们在两个数据集上验证了我们的方法：一个是使用现成的WiFi设备和摄像头收集的宽带CSI数据集；另一个是公开可用的MM-Fi数据子集。实验结果表明，在计算效率和感知质量方面，LatentCSI优于直接在真实图像上训练的复杂度相当的基础模型，同时通过其独特的文本引导可控性提供了实用优势。
URL
https://arxiv.org/abs/2506.10605
PDF
-
Only-Style: Stylistic Consistency in Image Generation without Content Leakage
2025-06-11 16:33:09Tilemachos Aravanis (School of Electrical & Computer Engineering, National Technical University of Athens, Greece), Panagiotis Filntisis (Robotics Institute, Athena Research Center, Maroussi, Greece, HERON - Center of Excellence in Robotics, Athens, Greece), Petros Maragos (School of Electrical & Computer Engineering, National Technical University of Athens, Greece, Robotics Institute, Athena Research Center, Maroussi, Greece, HERON - Center of Excellence in Robotics, Athens, Greece), George Retsinas (Robotics Institute, Athena Research Center, Maroussi, Greece, HERON - Center of Excellence in Robotics, Athens, Greece)Abstract
Generating images in a consistent reference visual style remains a challenging computer vision task. State-of-the-art methods aiming for style-consistent generation struggle to effectively separate semantic content from stylistic elements, leading to content leakage from the image provided as a reference to the targets. To address this challenge, we propose Only-Style: a method designed to mitigate content leakage in a semantically coherent manner while preserving stylistic consistency. Only-Style works by localizing content leakage during inference, allowing the adaptive tuning of a parameter that controls the style alignment process, specifically within the image patches containing the subject in the reference image. This adaptive process best balances stylistic consistency with leakage elimination. Moreover, the localization of content leakage can function as a standalone component, given a reference-target image pair, allowing the adaptive tuning of any method-specific parameter that provides control over the impact of the stylistic reference. In addition, we propose a novel evaluation framework to quantify the success of style-consistent generations in avoiding undesired content leakage. Our approach demonstrates a significant improvement over state-of-the-art methods through extensive evaluation across diverse instances, consistently achieving robust stylistic consistency without undesired content leakage.
Abstract (translated)
在一致的参考视觉风格下生成图像仍然是一个具有挑战性的计算机视觉任务。最先进的旨在实现风格一致性生成的方法难以有效地区分语义内容和风格元素，导致从提供的参考图片中向目标泄露不必要的内容信息。为了解决这一挑战，我们提出了Only-Style：一种方法，旨在以语义连贯的方式减少内容泄漏，同时保持风格的一致性。Only-Style通过在推理过程中定位内容泄漏来工作，并允许自适应调整控制样式对齐过程的参数（特别是在参考图像中包含主体的图块内）。这种自适应处理最能平衡风格一致性与泄漏消除之间的关系。此外，定位内容泄露的功能可以作为独立组件存在，给定一个参考目标图片对，允许任何特定方法的参数进行自适应调节，从而控制样式参照的影响。 我们还提出了一种新的评估框架，以量化风格一致生成在避免不需要的内容泄漏方面的成功程度。通过广泛的测试和各种实例，我们的方法明显优于最先进的方法，在保持强大风格一致性的同时，始终能有效地防止不需要的内容泄露。
URL
https://arxiv.org/abs/2506.09916
PDF
-
TuneGenie: Reasoning-based LLM agents for preferential music generation
2025-06-10 08:39:37Amitesh Pandey, Jafarbek Arifdjanov, Ansh TiwariAbstract
Recently, Large language models (LLMs) have shown great promise across a diversity of tasks, ranging from generating images to reasoning spatially. Considering their remarkable (and growing) textual reasoning capabilities, we investigate LLMs' potency in conducting analyses of an individual's preferences in music (based on playlist metadata, personal write-ups, etc.) and producing effective prompts (based on these analyses) to be passed to Suno AI (a generative AI tool for music production). Our proposition of a novel LLM-based textual representation to music model (which we call TuneGenie) and the various methods we develop to evaluate & benchmark similar models add to the increasing (and increasingly controversial) corpus of research on the use of AI in generating art.
Abstract (translated)
最近，大型语言模型（LLM）在多种任务中展现了巨大的潜力，从生成图像到进行空间推理。鉴于它们卓越且不断增长的文本推理能力，我们研究了这些模型在分析个人音乐偏好方面的效力（基于播放列表元数据、个人评论等），并开发有效提示（基于这些分析）以传递给Suno AI（一种用于音乐制作的生成式AI工具）。我们提出了一种新颖的LLM基文本表示到音乐模型（称之为TuneGenie），以及我们为评估和基准测试类似模型而发展的各种方法，都增加了关于在艺术创作中使用AI的研究日益增长且日益具有争议性的文献。
URL
https://arxiv.org/abs/2506.12083
PDF
-
OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation
2025-06-09 17:50:21Jingjing Chang, Yixiao Fang, Peng Xing, Shuhan Wu, Wei Cheng, Rui Wang, Xianfang Zeng, Gang Yu, Hai-Bao ChenAbstract
Text-to-image (T2I) models have garnered significant attention for generating high-quality images aligned with text prompts. However, rapid T2I model advancements reveal limitations in early benchmarks, lacking comprehensive evaluations, for example, the evaluation on reasoning, text rendering and style. Notably, recent state-of-the-art models, with their rich knowledge modeling capabilities, show promising results on the image generation problems requiring strong reasoning ability, yet existing evaluation systems have not adequately addressed this frontier. To systematically address these gaps, we introduce OneIG-Bench, a meticulously designed comprehensive benchmark framework for fine-grained evaluation of T2I models across multiple dimensions, including prompt-image alignment, text rendering precision, reasoning-generated content, stylization, and diversity. By structuring the evaluation, this benchmark enables in-depth analysis of model performance, helping researchers and practitioners pinpoint strengths and bottlenecks in the full pipeline of image generation. Specifically, OneIG-Bench enables flexible evaluation by allowing users to focus on a particular evaluation subset. Instead of generating images for the entire set of prompts, users can generate images only for the prompts associated with the selected dimension and complete the corresponding evaluation accordingly. Our codebase and dataset are now publicly available to facilitate reproducible evaluation studies and cross-model comparisons within the T2I research community.
Abstract (translated)
文本到图像（T2I）模型因其能生成高质量并与文本提示相匹配的图像而备受关注。然而，随着T2I模型的快速进步，早期基准测试中出现了一些局限性，例如缺乏全面评估，特别是在推理、文本渲染和风格方面。值得注意的是，最近最先进的模型在需要强大推理能力的图像生成问题上展现了令人鼓舞的结果，但现有的评估系统尚未充分解决这一领域的问题。为了系统地填补这些空白，我们引入了OneIG-Bench，这是一个精心设计的基准框架，旨在对T2I模型进行细致全面的多维度评估，包括提示-图像匹配度、文本渲染精度、推理生成内容、风格化和多样性等方面。通过结构化的评估方法，该基准能够深入分析模型性能，帮助研究人员和实践者识别整个图像生成流程中的优势与瓶颈。具体而言，OneIG-Bench允许用户灵活地进行特定评估子集的测试，而非为所有提示生成图像，使用者只需针对选定维度的相关提示生成图像，并完成相应的评估即可。我们的代码库和数据集现已公开发布，便于T2I研究社区内的可重复性研究及跨模型比较。
URL
https://arxiv.org/abs/2506.07977
PDF
-
Generalized Diffusion Detector: Mining Robust Features from Diffusion Models for Domain-Generalized Detection
2025-06-04 16:55:35Boyong He, Yuxiang Ji, Qianwen Ye, Zhuoyue Tan, Liaoni WuAbstract
Domain generalization (DG) for object detection aims to enhance detectors' performance in unseen scenarios. This task remains challenging due to complex variations in real-world applications. Recently, diffusion models have demonstrated remarkable capabilities in diverse scene generation, which inspires us to explore their potential for improving DG tasks. Instead of generating images, our method extracts multi-step intermediate features during the diffusion process to obtain domain-invariant features for generalized detection. Furthermore, we propose an efficient knowledge transfer framework that enables detectors to inherit the generalization capabilities of diffusion models through feature and object-level alignment, without increasing inference time. We conduct extensive experiments on six challenging DG benchmarks. The results demonstrate that our method achieves substantial improvements of 14.0% mAP over existing DG approaches across different domains and corruption types. Notably, our method even outperforms most domain adaptation methods without accessing any target domain data. Moreover, the diffusion-guided detectors show consistent improvements of 15.9% mAP on average compared to the baseline. Our work aims to present an effective approach for domain-generalized detection and provide potential insights for robust visual recognition in real-world scenarios. The code is available at this https URL.
Abstract (translated)
领域泛化（Domain Generalization，DG）在目标检测中的应用旨在提高检测器在未见过场景下的表现。由于实际应用场景中存在复杂多变的情况，这一任务依然充满挑战。最近，扩散模型展示了其在生成多样化场景方面的卓越能力，这激发了我们将它们用于改善DG任务的潜力的研究兴趣。与直接生成图像不同，我们的方法通过在扩散过程中提取多步中间特征来获取跨域不变性特征，从而实现泛化检测。 此外，我们提出了一种高效的知识转移框架，使检测器能够继承扩散模型的泛化能力，而无需增加推理时间，并且该框架是通过特征级和对象级对齐完成的。我们在六个具有挑战性的DG基准数据集上进行了广泛的实验，结果显示我们的方法在不同域和损坏类型上比现有DG方法提高了14.0% mAP（平均精度）。值得注意的是，即使不访问目标域数据，我们提出的方法也超越了大多数领域适应方法的表现。 此外，由扩散模型指导的检测器相比基准线，在所有测试场景中平均mAP值提升达到了15.9%。我们的工作旨在提供一种有效的跨领域泛化检测解决方案，并为在实际场景中的稳健视觉识别提供潜在见解。代码可以在以下链接获取：[此链接](https://this https URL)。 请注意，最后一个URL应替换为您实际提供的代码仓库地址或其他相关资源的正确链接。
URL
https://arxiv.org/abs/2503.02101
PDF
-
PointT2I: LLM-based text-to-image generation via keypoints
2025-06-02 06:55:20Taekyung Lee, Donggyu Lee, Myungjoo KangAbstract
Text-to-image (T2I) generation model has made significant advancements, resulting in high-quality images aligned with an input prompt. However, despite T2I generation's ability to generate fine-grained images, it still faces challenges in accurately generating images when the input prompt contains complex concepts, especially human pose. In this paper, we propose PointT2I, a framework that effectively generates images that accurately correspond to the human pose described in the prompt by using a large language model (LLM). PointT2I consists of three components: Keypoint generation, Image generation, and Feedback system. The keypoint generation uses an LLM to directly generate keypoints corresponding to a human pose, solely based on the input prompt, without external references. Subsequently, the image generation produces images based on both the text prompt and the generated keypoints to accurately reflect the target pose. To refine the outputs of the preceding stages, we incorporate an LLM-based feedback system that assesses the semantic consistency between the generated contents and the given prompts. Our framework is the first approach to leveraging LLM for keypoints-guided image generation without any fine-tuning, producing accurate pose-aligned images based solely on textual prompts.
Abstract (translated)
文本到图像（T2I）生成模型已经取得了显著进展，能够根据输入提示生成高质量的图像。然而，尽管T2I生成模型能够在某些方面产生细节丰富的图像，当输入提示包含复杂的概念时，尤其是人体姿态时，仍然面临准确生成图像的挑战。 在本文中，我们提出了PointT2I框架，该框架利用大规模语言模型（LLM）有效地生成与提示中描述的人体姿态相匹配的高质量图像。PointT2I包括三个组成部分：关键点生成、图像生成和反馈系统。 1. **关键点生成**：这部分使用一个大模型直接根据输入提示生成对应人体姿势的关键点，而不依赖于外部参考。 2. **图像生成**：该部分基于文本提示以及由前面步骤产生的关键点来生成图像，以准确反映目标姿态。 3. **反馈系统**：为了优化前两个阶段的输出，我们引入了一个基于大模型的反馈机制，它评估生成内容与给定提示之间的语义一致性。 我们的框架是首个不进行微调即可利用LLM引导关键点并据此生成精确对齐的人体姿势图像的方法。仅通过文本提示，PointT2I就能产生准确的姿态对齐图像。
URL
https://arxiv.org/abs/2506.01370
PDF
-
ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RL
2025-05-30 17:59:48Yu Zhang, Yunqi Li, Yifan Yang, Rui Wang, Yuqing Yang, Dai Qi, Jianmin Bao, Dongdong Chen, Chong Luo, Lili QiuAbstract
Although chain-of-thought reasoning and reinforcement learning (RL) have driven breakthroughs in NLP, their integration into generative vision models remains underexplored. We introduce ReasonGen-R1, a two-stage framework that first imbues an autoregressive image generator with explicit text-based "thinking" skills via supervised fine-tuning on a newly generated reasoning dataset of written rationales, and then refines its outputs using Group Relative Policy Optimization. To enable the model to reason through text before generating images, We automatically generate and release a corpus of model crafted rationales paired with visual prompts, enabling controlled planning of object layouts, styles, and scene compositions. Our GRPO algorithm uses reward signals from a pretrained vision language model to assess overall visual quality, optimizing the policy in each update. Evaluations on GenEval, DPG, and the T2I benchmark demonstrate that ReasonGen-R1 consistently outperforms strong baselines and prior state-of-the-art models. More: this http URL.
Abstract (translated)
尽管基于链式思维推理和强化学习（RL）在自然语言处理（NLP）领域取得了突破性进展，但它们在生成视觉模型中的整合仍处于探索阶段。我们引入了ReasonGen-R1框架，这是一个两阶段的架构：首先通过在新生成的推理数据集上进行监督微调，赋予自回归图像生成器显式的基于文本的“思考”技能；然后利用群体相对策略优化（GRPO）来改进其输出结果。为了使模型能够在生成图像之前通过文本进行推理，我们自动创建并发布了一套由模型创作的论述文集，这些文档与视觉提示配对，从而能够控制物体布局、风格和场景构图的设计。我们的GRPO算法使用预训练的视觉语言模型提供的奖励信号来评估整体视觉质量，并在每次更新中优化策略。在GenEval、DPG以及T2I基准测试上的评估结果表明，ReasonGen-R1始终优于强大的基线模型及先前最先进的模型。更多信息，请参见：[此链接](http://this.http.url)。 注意：原文中的“this http URL”是一个占位符或伪链接，并非实际链接地址，在正式发布时应替换为有效的链接地址。
URL
https://arxiv.org/abs/2505.24875
PDF
-
Unleashing High-Quality Image Generation in Diffusion Sampling Using Second-Order Levenberg-Marquardt-Langevin
2025-05-30 05:21:44Fangyikang Wang, Hubery Yin, Lei Qian, Yinan Li, Shaobin Zhuang, Huminhao Zhu, Yilin Zhang, Yanlong Tang, Chao Zhang, Hanbin Zhao, Hui Qian, Chen LiAbstract
The diffusion models (DMs) have demonstrated the remarkable capability of generating images via learning the noised score function of data distribution. Current DM sampling techniques typically rely on first-order Langevin dynamics at each noise level, with efforts concentrated on refining inter-level denoising strategies. While leveraging additional second-order Hessian geometry to enhance the sampling quality of Langevin is a common practice in Markov chain Monte Carlo (MCMC), the naive attempts to utilize Hessian geometry in high-dimensional DMs lead to quadratic-complexity computational costs, rendering them non-scalable. In this work, we introduce a novel Levenberg-Marquardt-Langevin (LML) method that approximates the diffusion Hessian geometry in a training-free manner, drawing inspiration from the celebrated Levenberg-Marquardt optimization algorithm. Our approach introduces two key innovations: (1) A low-rank approximation of the diffusion Hessian, leveraging the DMs' inherent structure and circumventing explicit quadratic-complexity computations; (2) A damping mechanism to stabilize the approximated Hessian. This LML approximated Hessian geometry enables the diffusion sampling to execute more accurate steps and improve the image generation quality. We further conduct a theoretical analysis to substantiate the approximation error bound of low-rank approximation and the convergence property of the damping mechanism. Extensive experiments across multiple pretrained DMs validate that the LML method significantly improves image generation quality, with negligible computational overhead.
Abstract (translated)
扩散模型（DM）通过学习数据分布的噪声得分函数，展示了生成图像的强大能力。目前的DM采样技术通常依赖于每个噪声水平上的第一阶Langevin动力学，并且努力集中在改进跨层级去噪策略上。尽管在马尔可夫链蒙特卡洛（MCMC）方法中利用额外的二阶Hessian几何来增强Langevin采样的质量是一种常见做法，但在高维DM中直接使用Hessian几何会导致二次复杂度的计算成本，使其无法扩展。 在这项工作中，我们引入了一种新的莱文伯格-马夸尔特-兰热姆（Levenberg-Marquardt-Langevin, LML）方法，该方法以无训练的方式近似扩散Hessian几何，借鉴了著名的Levenberg-Marquardt优化算法。我们的方法提出了两项关键创新： 1. 一种对扩散Hessian的低秩近似，利用DM固有的结构，并避免直接进行二次复杂度计算。 2. 一个阻尼机制来稳定近似的Hessian。 这种LML近似Hessian几何使扩散采样能够执行更准确的步骤并提高图像生成质量。我们进一步进行了理论分析以验证低秩近似误差界和阻尼机制的收敛性。在多个预训练DM上的广泛实验验证了LML方法显著提高了图像生成的质量，且计算开销几乎可以忽略不计。
URL
https://arxiv.org/abs/2505.24222
PDF
-
ComposeAnything: Composite Object Priors for Text-to-Image Generation
2025-05-30 00:13:36Zeeshan Khan, Shizhe Chen, Cordelia SchmidAbstract
Generating images from text involving complex and novel object arrangements remains a significant challenge for current text-to-image (T2I) models. Although prior layout-based methods improve object arrangements using spatial constraints with 2D layouts, they often struggle to capture 3D positioning and sacrifice quality and coherence. In this work, we introduce ComposeAnything, a novel framework for improving compositional image generation without retraining existing T2I models. Our approach first leverages the chain-of-thought reasoning abilities of LLMs to produce 2.5D semantic layouts from text, consisting of 2D object bounding boxes enriched with depth information and detailed captions. Based on this layout, we generate a spatial and depth aware coarse composite of objects that captures the intended composition, serving as a strong and interpretable prior that replaces stochastic noise initialization in diffusion-based T2I models. This prior guides the denoising process through object prior reinforcement and spatial-controlled denoising, enabling seamless generation of compositional objects and coherent backgrounds, while allowing refinement of inaccurate priors. ComposeAnything outperforms state-of-the-art methods on the T2I-CompBench and NSR-1K benchmarks for prompts with 2D/3D spatial arrangements, high object counts, and surreal compositions. Human evaluations further demonstrate that our model generates high-quality images with compositions that faithfully reflect the text.
Abstract (translated)
从文本生成包含复杂和新颖对象排列的图像，对于当前的文字到图像（T2I）模型来说仍是一个重大挑战。虽然以前基于布局的方法通过使用二维布局中的空间约束来改进对象排列，但它们常常难以捕捉三维定位，并且会牺牲质量和连贯性。在这项工作中，我们引入了ComposeAnything，这是一个新颖的框架，在不重新训练现有T2I模型的情况下，可以提高图像生成的组合能力。 我们的方法首先利用大语言模型（LLM）的链式思维推理能力，从文本中产生包含深度信息和详细说明的二维对象边界框的2.5D语义布局。基于这种布局，我们根据空间和深度感知生成一个粗略的复合图，该图捕获了预期的组合，并作为强大的、可解释的先验条件替代扩散型T2I模型中的随机噪声初始化。 这个先验通过物体优先级强化和空间控制去噪指导降噪过程，使复杂对象和连贯背景的无缝生成成为可能，同时允许对不准确的先验进行细化。ComposeAnything在具有二维/三维空间布局、高对象计数和超现实组合提示的T2I-CompBench和NSR-1K基准测试中超越了最先进的方法。 人类评估进一步表明，我们的模型能够生成高质量且忠实反映文本内容组成的图像。
URL
https://arxiv.org/abs/2505.24086
PDF
-
Rhetorical Text-to-Image Generation via Two-layer Diffusion Policy Optimization
2025-05-28 19:03:37Yuxi Zhang, Yueting Li, Xinyu Du, Sibo WangAbstract
Generating images from rhetorical languages remains a critical challenge for text-to-image models. Even state-of-the-art (SOTA) multimodal large language models (MLLM) fail to generate images based on the hidden meaning inherent in rhetorical language--despite such content being readily mappable to visual representations by humans. A key limitation is that current models emphasize object-level word embedding alignment, causing metaphorical expressions to steer image generation towards their literal visuals and overlook the intended semantic meaning. To address this, we propose Rhet2Pix, a framework that formulates rhetorical text-to-image generation as a multi-step policy optimization problem, incorporating a two-layer MDP diffusion module. In the outer layer, Rhet2Pix converts the input prompt into incrementally elaborated sub-sentences and executes corresponding image-generation actions, constructing semantically richer visuals. In the inner layer, Rhet2Pix mitigates reward sparsity during image generation by discounting the final reward and optimizing every adjacent action pair along the diffusion denoising trajectory. Extensive experiments demonstrate the effectiveness of Rhet2Pix in rhetorical text-to-image generation. Our model outperforms SOTA MLLMs such as GPT-4o, Grok-3 and leading academic baselines across both qualitative and quantitative evaluations. The code and dataset used in this work are publicly available.
Abstract (translated)
从修辞语言生成图像仍然是文本到图像模型面临的关键挑战。即使是最先进的多模态大型语言模型（MLLM）也无法根据修辞语言中固有的含义生成图像——尽管这些内容可以很容易地由人类映射为视觉表示形式。目前的模型主要侧重于对象级别的词嵌入对齐，这会导致比喻表达被导向其字面意义的视觉效果，并忽略所意图的意义。为了应对这一挑战，我们提出了Rhet2Pix框架，该框架将修辞文本到图像生成问题定义为一个多步骤策略优化问题，并引入了一个两层MDP扩散模块。 在外层，Rhet2Pix将输入提示转换为逐步详细化的子句并执行相应的图像生成操作，以此构建语义更丰富的视觉效果。在内层，Rhet2Pix通过减少最终奖励并沿扩散去噪轨迹优化每一个相邻的动作对来缓解图像生成过程中的奖励稀疏性问题。 大量的实验展示了Rhet2Pix在修辞文本到图像生成方面的有效性。我们的模型在定性和定量评估中均超过了最先进的MLLM（如GPT-4o，GroK-3）和学术基准模型。本研究使用的代码和数据集已公开提供。
URL
https://arxiv.org/abs/2505.22792
PDF
-
Policy Optimized Text-to-Image Pipeline Design
2025-05-27 17:50:47Uri Gadot, Rinon Gal, Yftah Ziser, Gal Chechik, Shie MannorAbstract
Text-to-image generation has evolved beyond single monolithic models to complex multi-component pipelines. These combine fine-tuned generators, adapters, upscaling blocks and even editing steps, leading to significant improvements in image quality. However, their effective design requires substantial expertise. Recent approaches have shown promise in automating this process through large language models (LLMs), but they suffer from two critical limitations: extensive computational requirements from generating images with hundreds of predefined pipelines, and poor generalization beyond memorized training examples. We introduce a novel reinforcement learning-based framework that addresses these inefficiencies. Our approach first trains an ensemble of reward models capable of predicting image quality scores directly from prompt-workflow combinations, eliminating the need for costly image generation during training. We then implement a two-phase training strategy: initial workflow vocabulary training followed by GRPO-based optimization that guides the model toward higher-performing regions of the workflow space. Additionally, we incorporate a classifier-free guidance based enhancement technique that extrapolates along the path between the initial and GRPO-tuned models, further improving output quality. We validate our approach through a set of comparisons, showing that it can successfully create new flows with greater diversity and lead to superior image quality compared to existing baselines.
Abstract (translated)
文本到图像的生成技术已经从单一的整体模型发展到了复杂的多组件流水线。这些流水线结合了经过微调的生成器、适配器、上采样模块以及编辑步骤，从而显著提升了图像质量。然而，这种流水线的有效设计需要大量的专业知识。最近的研究表明，通过大型语言模型（LLMs）可以实现自动化这一过程，但是这种方法存在两个关键限制：一是由于需要生成数百个预定义的流水线来完成任务而产生的巨大计算需求；二是对超出训练样本的记忆泛化能力较差。我们提出了一种基于强化学习的新框架，以解决这些效率问题。 我们的方法首先训练一组奖励模型，这些模型能够直接从提示-工作流组合中预测图像质量得分，从而在训练过程中消除了昂贵的图像生成需求。接下来，我们实施了两阶段的训练策略：首先是初始的工作流程词汇表训练，然后是基于GRPO（Guided Reinforcement Policy Optimization）的优化，该优化引导模型向表现更好的工作流程空间区域发展。此外，我们还引入了一种无需分类器指导的技术改进方法，通过在初始模型与经过GRPO调整后的模型之间的路径上进行外推来进一步提升输出质量。 为了验证我们的方法的有效性，我们通过一系列对比实验进行了测试，并展示了该方法能够成功创建多样化的新型工作流程，并且相比现有基准，在图像质量方面表现出色。
URL
https://arxiv.org/abs/2505.21478
PDF
