Image_Generation
-
M$^{3}$T2IBench: A Large-Scale Multi-Category, Multi-Instance, Multi-Relation Text-to-Image Benchmark
2025-10-27 05:32:50Huixuan Zhang, Xiaojun WanAbstract
Text-to-image models are known to struggle with generating images that perfectly align with textual prompts. Several previous studies have focused on evaluating image-text alignment in text-to-image generation. However, these evaluations either address overly simple scenarios, especially overlooking the difficulty of prompts with multiple different instances belonging to the same category, or they introduce metrics that do not correlate well with human evaluation. In this study, we introduce M$^3$T2IBench, a large-scale, multi-category, multi-instance, multi-relation along with an object-detection-based evaluation metric, $AlignScore$, which aligns closely with human evaluation. Our findings reveal that current open-source text-to-image models perform poorly on this challenging benchmark. Additionally, we propose the Revise-Then-Enforce approach to enhance image-text alignment. This training-free post-editing method demonstrates improvements in image-text alignment across a broad range of diffusion models. \footnote{Our code and data has been released in supplementary material and will be made publicly available after the paper is accepted.}
Abstract (translated)
URL
https://arxiv.org/abs/2510.23020
PDF
-
From Competition to Synergy: Unlocking Reinforcement Learning for Subject-Driven Image Generation
2025-10-21 03:32:26Ziwei Huang, Ying Shu, Hao Fang, Quanyu Long, Wenya Wang, Qiushi Guo, Tiezheng Ge, Leilei GanAbstract
Subject-driven image generation models face a fundamental trade-off between identity preservation (fidelity) and prompt adherence (editability). While online reinforcement learning (RL), specifically GPRO, offers a promising solution, we find that a naive application of GRPO leads to competitive degradation, as the simple linear aggregation of rewards with static weights causes conflicting gradient signals and a misalignment with the temporal dynamics of the diffusion process. To overcome these limitations, we propose Customized-GRPO, a novel framework featuring two key innovations: (i) Synergy-Aware Reward Shaping (SARS), a non-linear mechanism that explicitly penalizes conflicted reward signals and amplifies synergistic ones, providing a sharper and more decisive gradient. (ii) Time-Aware Dynamic Weighting (TDW), which aligns the optimization pressure with the model's temporal dynamics by prioritizing prompt-following in the early, identity preservation in the later. Extensive experiments demonstrate that our method significantly outperforms naive GRPO baselines, successfully mitigating competitive degradation. Our model achieves a superior balance, generating images that both preserve key identity features and accurately adhere to complex textual prompts.
Abstract (translated)
基于主题的图像生成模型在身份保真（fidelity）和指令遵循（editability）之间存在根本性的权衡。虽然在线强化学习（RL），特别是GPRO，提供了一个有前景的解决方案，但我们发现直接应用GRPO会导致竞争性退化，因为简单的线性奖励聚合使用静态权重导致了冲突的梯度信号，并且与扩散过程的时间动态不一致。为了解决这些问题，我们提出了Customized-GRPO，这是一个包含两个关键创新的新框架： (i) 协同感知奖励塑形（Synergy-Aware Reward Shaping, SARS）：这是一种非线性机制，它明确地惩罚冲突的奖励信号并放大协同作用的信号，提供更尖锐和更具决定性的梯度。 (ii) 时间感知动态加权（Time-Aware Dynamic Weighting, TDW）：通过优先考虑早期阶段遵循指令、晚期阶段保持身份特征，该方法使优化压力与模型的时间动态相一致。 广泛的实验表明，我们的方法显著优于简单的GRPO基线，成功地缓解了竞争性退化。我们的模型实现了更佳的平衡，生成出既能保留关键的身份特征又能准确遵循复杂文本提示的图像。
URL
https://arxiv.org/abs/2510.18263
PDF
-
Salient Concept-Aware Generative Data Augmentation
2025-10-16 23:31:55Tianchen Zhao, Xuanbai Chen, Zhihua Li, Jun Fang, Dongsheng An, Xiang Xu, Zhuowen Tu, Yifan XingAbstract
Recent generative data augmentation methods conditioned on both image and text prompts struggle to balance between fidelity and diversity, as it is challenging to preserve essential image details while aligning with varied text prompts. This challenge arises because representations in the synthesis process often become entangled with non-essential input image attributes such as environmental contexts, creating conflicts with text prompts intended to modify these elements. To address this, we propose a personalized image generation framework that uses a salient concept-aware image embedding model to reduce the influence of irrelevant visual details during the synthesis process, thereby maintaining intuitive alignment between image and text inputs. By generating images that better preserve class-discriminative features with additional controlled variations, our framework effectively enhances the diversity of training datasets and thereby improves the robustness of downstream models. Our approach demonstrates superior performance across eight fine-grained vision datasets, outperforming state-of-the-art augmentation methods with averaged classification accuracy improvements by 0.73% and 6.5% under conventional and long-tail settings, respectively.
Abstract (translated)
最近的基于图像和文本提示的生成数据增强方法在保真度与多样性之间难以取得平衡，因为这些方法往往很难在保留图像关键细节的同时适应多样化的文本提示。这一挑战源于合成过程中表示的纠缠问题，即重要图像属性（如核心视觉特征）经常与其他非本质输入属性（例如环境背景信息）混杂在一起，导致与旨在修改这些元素的文本提示产生冲突。 为解决上述难题，我们提出了一种个性化图像生成框架，该框架采用显著概念感知的图像嵌入模型，减少合成过程中无关视觉细节的影响。这样一来，可以更好地保持图像和文本输入之间直观一致的关系。通过在保留类别判别性特征的同时引入额外控制变化，我们的方法能够有效增加训练数据集的多样性，并提升下游模型的鲁棒性。 实验结果表明，在八个细粒度视觉数据集中，相较于现有先进增强技术，我们的方法表现出色：常规设置下分类准确率平均提高了0.73%，长尾分布场景下则提升了6.5%。
URL
https://arxiv.org/abs/2510.15194
PDF
-
GIR-Bench: Versatile Benchmark for Generating Images with Reasoning
2025-10-13 05:50:44Hongxiang Li, Yaowei Li, Bin Lin, Yuwei Niu, Yuhang Yang, Xiaoshuang Huang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Long ChenAbstract
Unified multimodal models integrate the reasoning capacity of large language models with both image understanding and generation, showing great promise for advanced multimodal intelligence. However, the community still lacks a rigorous reasoning-centric benchmark to systematically evaluate the alignment between understanding and generation, and their generalization potential in complex visual tasks. To this end, we introduce \textbf{GIR-Bench}, a comprehensive benchmark that evaluates unified models across three complementary perspectives. Firstly, we investigate understanding-generation consistency (GIR-Bench-UGC), asking whether models can consistently leverage the same knowledge in both understanding and generation tasks. Secondly, we investigate whether models can perform reasoning-centric text-to-image generation that requires applying logical constraints and implicit knowledge to generate faithful visual content (GIR-Bench-T2I). Thirdly, we evaluate whether models can handle multi-step reasoning in editing (GIR-Bench-Edit). For each subset, we carefully design different task-specific evaluation pipelines tailored for each task. This enables fine-grained and interpretable evaluation while mitigating biases from the prevalent MLLM-as-a-Judge paradigm. Extensive ablations over various unified models and generation-only systems have shown that: Although unified models are more capable of reasoning-driven visual tasks, they still exhibit a persistent gap between understanding and generation. The data and code for GIR-Bench are available at \href{this https URL}{this https URL}.
Abstract (translated)
统一的多模态模型将大型语言模型的推理能力与图像理解和生成相结合，展示了先进多模态智能的巨大潜力。然而，社区仍然缺乏一个以推理为中心的严格基准来系统地评估理解与生成之间的对齐情况以及它们在复杂视觉任务中的泛化潜力。为此，我们引入了**GIR-Bench**这一全面基准，它从三个方面综合评价统一模型：首先，我们探讨理解和生成的一致性（GIR-Bench-UGC），即模型是否能在理解和生成任务中一致地利用相同的知识；其次，我们研究模型能否执行以推理为中心的文本到图像生成，这需要应用逻辑约束和隐含知识来生成忠实的视觉内容（GIR-Bench-T2I）；最后，我们评估模型处理编辑中的多步推理的能力（GIR-Bench-Edit）。对于每个子集，我们都精心设计了不同的任务特定评估流程。这一方法使得评价更加细致入微且具有可解释性，并能减轻MLLM-as-a-Judge范式带来的偏见。 通过对各种统一模型和仅生成系统的广泛消融研究发现：尽管统一模型在推理驱动的视觉任务上更擅长，它们在理解和生成之间仍然存在持续性的差距。GIR-Bench的数据和代码可在[此处](this https URL)获取。
URL
https://arxiv.org/abs/2510.11026
PDF
-
The Digital Mirror: Gender Bias and Occupational Stereotypes in AI-Generated Images
2025-10-08 12:49:18Siiri Lepp\"alampi, Sonja M. Hyrynsalmi, Erno VanhalaAbstract
Generative AI offers vast opportunities for creating visualisations, such as graphics, videos, and images. However, recent studies around AI-generated visualisations have primarily focused on the creation process and image quality, overlooking representational biases. This study addresses this gap by testing representation biases in AI-generated pictures in an occupational setting and evaluating how two AI image generator tools, DALL-E 3 and Ideogram, compare. Additionally, the study discusses topics such as ageing and emotions in AI-generated images. As AI image tools are becoming more widely used, addressing and mitigating harmful gender biases becomes essential to ensure diverse representation in media and professional settings. In this study, over 750 AI-generated images of occupations were prompted. The thematic analysis results revealed that both DALL-E 3 and Ideogram reinforce traditional gender stereotypes in AI-generated images, although to varying degrees. These findings emphasise that AI visualisation tools risk reinforcing narrow representations. In our discussion section, we propose suggestions for practitioners, individuals and researchers to increase representation when generating images with visible genders.
Abstract (translated)
生成式AI为创建可视化内容（如图形、视频和图像）提供了巨大的机会。然而，最近关于AI生成的可视化研究主要集中在创作过程和图像质量上，而忽略了代表性偏见。本研究旨在填补这一空白，通过测试在职业环境中的AI生成图片的代表性偏见，并评估两种AI图像生成工具——DALL-E 3和Ideogram之间的差异。 此外，该研究还讨论了年龄和情感等主题在AI生成图像中的表现情况。随着AI图像工具被越来越广泛地使用，解决并缓解有害性别偏见变得至关重要，以确保媒体和专业环境中具有多样化的代表性。在这项研究中，触发了超过750张不同职业的AI生成图片。主题分析结果表明，DALL-E 3和Ideogram在生成包含明确性别的图像时都会强化传统的性别刻板印象，尽管程度有所不同。 这些发现强调了AI可视化工具可能会固化狭隘的表现形式的风险。在我们的讨论部分中，我们为从业者、个人以及研究人员提出了建议，旨在提高在生成具有可见性别的图像时的代表性多样性。
URL
https://arxiv.org/abs/2510.08628
PDF
-
GHOST: Hallucination-Inducing Image Generation for Multimodal LLMs
2025-09-29 17:59:23Aryan Yazdan Parast, Parsa Hosseini, Hesam Asadollahzadeh, Arshia Soltani Moakhar, Basim Azam, Soheil Feizi, Naveed AkhtarAbstract
Object hallucination in Multimodal Large Language Models (MLLMs) is a persistent failure mode that causes the model to perceive objects absent in the image. This weakness of MLLMs is currently studied using static benchmarks with fixed visual scenarios, which preempts the possibility of uncovering model-specific or unanticipated hallucination vulnerabilities. We introduce GHOST (Generating Hallucinations via Optimizing Stealth Tokens), a method designed to stress-test MLLMs by actively generating images that induce hallucination. GHOST is fully automatic and requires no human supervision or prior knowledge. It operates by optimizing in the image embedding space to mislead the model while keeping the target object absent, and then guiding a diffusion model conditioned on the embedding to generate natural-looking images. The resulting images remain visually natural and close to the original input, yet introduce subtle misleading cues that cause the model to hallucinate. We evaluate our method across a range of models, including reasoning models like GLM-4.1V-Thinking, and achieve a hallucination success rate exceeding 28%, compared to around 1% in prior data-driven discovery methods. We confirm that the generated images are both high-quality and object-free through quantitative metrics and human evaluation. Also, GHOST uncovers transferable vulnerabilities: images optimized for Qwen2.5-VL induce hallucinations in GPT-4o at a 66.5% rate. Finally, we show that fine-tuning on our images mitigates hallucination, positioning GHOST as both a diagnostic and corrective tool for building more reliable multimodal systems.
Abstract (translated)
多模态大型语言模型（MLLMs）中的对象幻觉是一种持续存在的失败模式，导致模型在图像中感知到实际不存在的对象。目前，研究这种模型弱点的方法主要是通过使用静态基准测试来完成的，这些基准测试包含固定的视觉场景，这可能无法揭示特定于某个模型或未被预见的幻觉漏洞。为此，我们引入了一种名为GHOST（通过优化隐蔽标记生成幻觉）的新方法，旨在通过主动生成诱导幻觉的图像对MLLMs进行压力测试。这种方法是全自动化的，并不需要人类监督或先验知识。 GHOST的工作原理是在不包含目标对象的情况下，在图像嵌入空间中进行优化以误导模型，然后引导基于该嵌入条件的扩散模型生成看起来自然的图像。生成的图像在视觉上仍然保持与原始输入相似和高质量的状态，但会引入一些微妙的误导性线索，导致模型产生幻觉。 我们通过一系列包括推理模型（如GLM-4.1V-Thinking）在内的不同模型对我们的方法进行了评估，并且达到了超过28%的幻觉成功率，相比之下，先前基于数据驱动的方法仅为大约1%。我们还通过定量指标和人类评价确认了生成的图像是高质量且无对象存在的。 此外，GHOST揭示了一种可迁移性漏洞：为Qwen2.5-VL优化的图像在GPT-4o中诱导幻觉的成功率为66.5%。最后，我们展示了对我们的图片进行微调可以减轻幻觉现象的发生率，从而将GHOST定位为一种诊断和纠正工具，有助于构建更可靠的多模态系统。
URL
https://arxiv.org/abs/2509.25178
PDF
-
Simulating Post-Neoadjuvant Chemotherapy Breast Cancer MRI via Diffusion Model with Prompt Tuning
2025-09-29 02:05:20Jonghun Kim, Hyunjin ParkAbstract
Neoadjuvant chemotherapy (NAC) is a common therapy option before the main surgery for breast cancer. Response to NAC is monitored using follow-up dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI). Accurate prediction of NAC response helps with treatment planning. Here, we adopt maximum intensity projection images from DCE-MRI to generate post-treatment images (i.e., 3 or 12 weeks after NAC) from pre-treatment images leveraging the emerging diffusion model. We introduce prompt tuning to account for the known clinical factors affecting response to NAC. Our model performed better than other generative models in image quality metrics. Our model was better at generating images that reflected changes in tumor size according to pCR compared to other models. Ablation study confirmed the design choices of our method. Our study has the potential to help with precision medicine.
Abstract (translated)
新辅助化疗（NAC）是乳腺癌主要手术前的常见治疗选项。通过随访的动态对比增强磁共振成像（DCE-MRI）来监测对NAC的反应。准确预测NAC的反应有助于制定治疗计划。在这里，我们采用来自DCE-MRI的最大强度投影图像，从预处理图像生成术后图像（即，NAC后的3周或12周），利用新兴的扩散模型实现这一点。我们引入提示调优来考虑已知影响NAC反应的临床因素。我们的模型在图像质量指标上优于其他生成模型，并且根据病理完全缓解（pCR）产生的肿瘤大小变化的图像比其他模型更为准确。消融研究证实了我们方法的设计选择。本研究有可能帮助实现精准医疗。
URL
https://arxiv.org/abs/2509.24185
PDF
-
LayoutAgent: A Vision-Language Agent Guided Compositional Diffusion for Spatial Layout Planning
2025-09-24 20:41:04Zezhong Fan, Xiaohan Li, Luyi Ma, Kai Zhao, Liang Peng, Topojoy Biswas, Evren Korpeoglu, Kaushiki Nag, Kannan AchanAbstract
Designing realistic multi-object scenes requires not only generating images, but also planning spatial layouts that respect semantic relations and physical plausibility. On one hand, while recent advances in diffusion models have enabled high-quality image generation, they lack explicit spatial reasoning, leading to unrealistic object layouts. On the other hand, traditional spatial planning methods in robotics emphasize geometric and relational consistency, but they struggle to capture semantic richness in visual scenes. To bridge this gap, in this paper, we propose LayoutAgent, an agentic framework that unifies vision-language reasoning with compositional diffusion for layout generation. Given multiple input images with target objects in them, our method first employs visual-language model to preprocess the inputs through segmentation, object size estimation, scene graph construction, and prompt rewriting. Then we leverage compositional diffusion-a method traditionally used in robotics-to synthesize bounding boxes that respect object relations encoded in the scene graph for spatial layouts. In the end, a foreground-conditioned image generator composes the complete scene by rendering the objects into the planned layout guided by designed prompts. Experiments demonstrate that LayoutAgent outperforms other state-of-the-art layout generation models in layout coherence, spatial realism and aesthetic alignment.
Abstract (translated)
设计逼真的多物体场景不仅需要生成图像，还需要规划符合语义关系和物理真实性的空间布局。一方面，尽管最近在扩散模型方面的进展已能够实现高质量的图像生成，但这些模型缺乏明确的空间推理能力，导致对象布局不现实。另一方面，传统的机器人领域中的空间规划方法强调几何和关联一致性，但在捕捉视觉场景中的语义丰富性方面存在困难。 为了解决这一问题，在本文中我们提出了LayoutAgent，这是一个将视觉-语言推理与组合扩散相结合的代理框架，用于生成布局。给定包含目标对象的多个输入图像后，我们的方法首先使用视觉-语言模型对这些输入进行预处理，包括分割、物体大小估计、场景图构建以及提示重写等步骤。然后，我们利用传统上在机器人领域中使用的组合扩散方法来合成符合场景图中的物体系数的空间布局边界框。最后，在设计的提示指导下，一个前景条件下的图像生成器通过将对象渲染到计划好的布局中来组成完整的场景。 实验结果表明，LayoutAgent在布局一致性、空间真实性和美学对齐方面优于其他最先进的布局生成模型。
URL
https://arxiv.org/abs/2509.22720
PDF
-
AcT2I: Evaluating and Improving Action Depiction in Text-to-Image Models
2025-09-19 16:41:39Vatsal Malaviya, Agneet Chatterjee, Maitreya Patel, Yezhou Yang, Chitta BaralAbstract
Text-to-Image (T2I) models have recently achieved remarkable success in generating images from textual descriptions. However, challenges still persist in accurately rendering complex scenes where actions and interactions form the primary semantic focus. Our key observation in this work is that T2I models frequently struggle to capture nuanced and often implicit attributes inherent in action depiction, leading to generating images that lack key contextual details. To enable systematic evaluation, we introduce AcT2I, a benchmark designed to evaluate the performance of T2I models in generating images from action-centric prompts. We experimentally validate that leading T2I models do not fare well on AcT2I. We further hypothesize that this shortcoming arises from the incomplete representation of the inherent attributes and contextual dependencies in the training corpora of existing T2I models. We build upon this by developing a training-free, knowledge distillation technique utilizing Large Language Models to address this limitation. Specifically, we enhance prompts by incorporating dense information across three dimensions, observing that injecting prompts with temporal details significantly improves image generation accuracy, with our best model achieving an increase of 72%. Our findings highlight the limitations of current T2I methods in generating images that require complex reasoning and demonstrate that integrating linguistic knowledge in a systematic way can notably advance the generation of nuanced and contextually accurate images.
Abstract (translated)
最近，文本到图像（Text-to-Image，T2I）模型在从文字描述生成图片方面取得了显著的成功。然而，在渲染以动作和互动为主要语义焦点的复杂场景时，依然存在挑战。我们在这项工作中的关键观察是：T2I模型经常难以捕捉行动描绘中内在的细微且常常隐含的特点，导致生成的图像缺少重要的上下文细节。 为了进行系统的评估，我们引入了AcT2I这一基准测试平台，旨在评价T2I模型在处理以动作为中心提示时的表现。通过实验验证，我们发现领先的T2I模型在AcT2I上的表现不尽如人意。进一步地，我们假设这种不足源于现有T2I模型训练数据中内在属性和上下文依赖关系的不完整表示。 为了解决这一限制，我们开发了一种无需重新训练的知识蒸馏技术，利用大型语言模型进行改进。具体而言，通过在提示语中加入跨三个维度的密集信息，特别是增加时间细节，显著提高了图像生成的准确性。我们的最佳模型在此方法下实现了72%的性能提升。 这些发现揭示了目前T2I方法在生成需要复杂推理的图片时存在的局限性，并表明系统地整合语言知识可以明显提高生成细腻且上下文准确度高的图像的能力。
URL
https://arxiv.org/abs/2509.16141
PDF
-
Iterative Prompt Refinement for Safer Text-to-Image Generation
2025-09-17 07:16:06Jinwoo Jeon, JunHyeok Oh, Hayeong Lee, Byung-Jun LeeAbstract
Text-to-Image (T2I) models have made remarkable progress in generating images from text prompts, but their output quality and safety still depend heavily on how prompts are phrased. Existing safety methods typically refine prompts using large language models (LLMs), but they overlook the images produced, which can result in unsafe outputs or unnecessary changes to already safe prompts. To address this, we propose an iterative prompt refinement algorithm that uses Vision Language Models (VLMs) to analyze both the input prompts and the generated images. By leveraging visual feedback, our method refines prompts more effectively, improving safety while maintaining user intent and reliability comparable to existing LLM-based approaches. Additionally, we introduce a new dataset labeled with both textual and visual safety signals using off-the-shelf multi-modal LLM, enabling supervised fine-tuning. Experimental results demonstrate that our approach produces safer outputs without compromising alignment with user intent, offering a practical solution for generating safer T2I content. Our code is available at this https URL. \textbf{\textcolor{red}WARNING: This paper contains examples of harmful or inappropriate images generated by models.
Abstract (translated)
文本到图像（T2I）模型在从文字提示生成图像方面取得了显著进展，但其输出质量和安全性仍然很大程度上取决于提示语的措辞。现有的安全方法通常利用大型语言模型（LLMs）来优化提示语，但它们忽略了生成的图片本身，这可能导致不安全的输出或对原本已经安全的提示进行不必要的修改。为了应对这一挑战，我们提出了一种迭代式提示优化算法，该算法使用视觉语言模型（VLMs）来分析输入提示和生成的图像两者。通过利用视觉反馈，我们的方法能够更有效地优化提示语，在提高安全性的同时保持用户意图并维持与现有LLM基线方法相当的可靠性。此外，我们还引入了一个新数据集，其中文本和视觉安全信号均使用现成的多模态语言模型标注，从而允许进行监督微调。实验结果表明，我们的方法能够在不牺牲与用户意图的一致性的情况下生成更安全的内容，为生成更安全的T2I内容提供了一种实用方案。我们的代码可在以下网址获得：[此URL]。 **警告：该论文包含由模型生成的有害或不当图像的例子。** 请注意，在处理这类研究时需要格外小心，并确保在适当的安全和伦理框架内操作这些数据，以避免任何潜在的危害。
URL
https://arxiv.org/abs/2509.13760
PDF
-
InfGen: A Resolution-Agnostic Paradigm for Scalable Image Synthesis
2025-09-12 17:48:57Tao Han, Wanghan Xu, Junchao Gong, Xiaoyu Yue, Song Guo, Luping Zhou, Lei BaiAbstract
Arbitrary resolution image generation provides a consistent visual experience across devices, having extensive applications for producers and consumers. Current diffusion models increase computational demand quadratically with resolution, causing 4K image generation delays over 100 seconds. To solve this, we explore the second generation upon the latent diffusion models, where the fixed latent generated by diffusion models is regarded as the content representation and we propose to decode arbitrary resolution images with a compact generated latent using a one-step generator. Thus, we present the \textbf{InfGen}, replacing the VAE decoder with the new generator, for generating images at any resolution from a fixed-size latent without retraining the diffusion models, which simplifies the process, reducing computational complexity and can be applied to any model using the same latent space. Experiments show InfGen is capable of improving many models into the arbitrary high-resolution era while cutting 4K image generation time to under 10 seconds.
Abstract (translated)
任意分辨率图像生成为不同设备提供了一致的视觉体验，在生产和消费领域有着广泛的应用。当前的扩散模型（diffusion models）计算需求随着分辨率的增加呈二次增长，导致4K图像生成延迟超过100秒。为了应对这一问题，我们探索了基于潜在扩散模型的第二代技术，其中由扩散模型产生的固定潜在表示被视为内容表示，并且我们提出使用一步式生成器从紧凑的潜在表示中解码任意分辨率的图像。因此，我们提出了**InfGen**方法，通过用新的生成器替换变分自编码器（VAE）解码器，在不重新训练扩散模型的情况下，可以从固定大小的潜在表示生成任何分辨率的图像，简化了流程并减少了计算复杂性，适用于使用相同潜在空间的任何模型。实验表明，InfGen能够将许多模型提升到任意高分辨率时代，并且将4K图像生成时间缩短至10秒以内。
URL
https://arxiv.org/abs/2509.10441
PDF
-
Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesis
2025-09-11 16:34:57Yikang Ding, Jiwen Liu, Wenyuan Zhang, Zekun Wang, Wentao Hu, Liyuan Cui, Mingming Lao, Yingchao Shao, Hui Liu, Xiaohan Li, Ming Chen, Xiaoqiang Liu, Yu-Shen Liu, Pengfei WanAbstract
Recent advances in audio-driven avatar video generation have significantly enhanced audio-visual realism. However, existing methods treat instruction conditioning merely as low-level tracking driven by acoustic or visual cues, without modeling the communicative purpose conveyed by the instructions. This limitation compromises their narrative coherence and character expressiveness. To bridge this gap, we introduce Kling-Avatar, a novel cascaded framework that unifies multimodal instruction understanding with photorealistic portrait generation. Our approach adopts a two-stage pipeline. In the first stage, we design a multimodal large language model (MLLM) director that produces a blueprint video conditioned on diverse instruction signals, thereby governing high-level semantics such as character motion and emotions. In the second stage, guided by blueprint keyframes, we generate multiple sub-clips in parallel using a first-last frame strategy. This global-to-local framework preserves fine-grained details while faithfully encoding the high-level intent behind multimodal instructions. Our parallel architecture also enables fast and stable generation of long-duration videos, making it suitable for real-world applications such as digital human livestreaming and vlogging. To comprehensively evaluate our method, we construct a benchmark of 375 curated samples covering diverse instructions and challenging scenarios. Extensive experiments demonstrate that Kling-Avatar is capable of generating vivid, fluent, long-duration videos at up to 1080p and 48 fps, achieving superior performance in lip synchronization accuracy, emotion and dynamic expressiveness, instruction controllability, identity preservation, and cross-domain generalization. These results establish Kling-Avatar as a new benchmark for semantically grounded, high-fidelity audio-driven avatar synthesis.
Abstract (translated)
最近在音频驱动的虚拟化身视频生成方面取得的进步显著提升了视听真实感。然而，现有的方法仅仅将指令条件化视为由声学或视觉线索驱动的基本级跟踪，并没有建模指令传达的沟通意图。这种局限性损害了它们的故事连贯性和角色表现力。为了解决这一问题，我们提出了Kling-Avatar，这是一种新的级联框架，它统一了多模式指令理解和逼真的肖像生成。 我们的方法采用了两阶段管道。在第一阶段中，我们设计了一种多模态大型语言模型（MLLM）导演，该导演根据各种指令信号生成蓝图视频，从而控制角色动作和情感等高层次语义。在第二阶段中，在蓝本关键帧的引导下，我们使用首尾帧策略并行生成多个子片段。这种从全局到局部的方法在保持细粒度细节的同时，忠实地编码了多模式指令背后的高层次意图。 我们的并行架构还能够快速而稳定地生成长时间视频，使其适用于数字人类直播和Vlogging等现实世界应用中。为了全面评估我们方法的效果，我们构建了一个基准测试集，包括375个精心挑选的样本，涵盖了各种各样的指令和挑战性的场景。广泛的实验表明，Kling-Avatar能够以高达1080p和48fps的速度生成生动、流畅、长时间视频，在唇同步准确度、情感及动态表现力、指令可控性、身份保持以及跨域泛化能力方面表现出色。 这些结果使Kling-Avatar成为语义支持的高保真音频驱动虚拟化身合成的新基准。
URL
https://arxiv.org/abs/2509.09595
PDF
-
Feature Space Analysis by Guided Diffusion Model
2025-09-09 17:18:39Kimiaki Shirahama, Miki Yanobu, Kaduki Yamashita, Miho OhsakiAbstract
One of the key issues in Deep Neural Networks (DNNs) is the black-box nature of their internal feature extraction process. Targeting vision-related domains, this paper focuses on analysing the feature space of a DNN by proposing a decoder that can generate images whose features are guaranteed to closely match a user-specified feature. Owing to this guarantee that is missed in past studies, our decoder allows us to evidence which of various attributes in an image are encoded into a feature by the DNN, by generating images whose features are in proximity to that feature. Our decoder is implemented as a guided diffusion model that guides the reverse image generation of a pre-trained diffusion model to minimise the Euclidean distance between the feature of a clean image estimated at each step and the user-specified feature. One practical advantage of our decoder is that it can analyse feature spaces of different DNNs with no additional training and run on a single COTS GPU. The experimental results targeting CLIP's image encoder, ResNet-50 and vision transformer demonstrate that images generated by our decoder have features remarkably similar to the user-specified ones and reveal valuable insights into these DNNs' feature spaces.
Abstract (translated)
深度神经网络（DNN）的一个关键问题是其内部特征提取过程的黑盒性质。针对视觉相关领域，本文通过提出一种解码器来分析DNN的特征空间，该解码器可以生成具有与用户指定特性紧密匹配的图像特征的图像。由于以往的研究中忽略了这一保证，我们的解码器能够生成接近特定特性的图像，从而揭示各种属性在图像中的编码情况。我们的解码器实现为一种引导扩散模型，它指导预训练的扩散模型进行逆向图像生成过程，以最小化每一步估计的清洁图像特征与用户指定特征之间的欧氏距离。 我们解码器的一个实际优势是它可以无需额外训练地分析不同DNN的特征空间，并且可以在单个商用现成（COTS）GPU上运行。针对CLIP的图像编码器、ResNet-50和视觉变压器进行的实验结果表明，由我们的解码器生成的图像具有与用户指定特性非常相似的特征，并揭示了这些DNN特征空间中的宝贵见解。
URL
https://arxiv.org/abs/2509.07936
PDF
-
Evaluating and comparing gender bias across four text-to-image models
2025-09-07 22:15:58Zoya Hammad, Nii Longdon SowahAbstract
As we increasingly use Artificial Intelligence (AI) in decision-making for industries like healthcare, finance, e-commerce, and even entertainment, it is crucial to also reflect on the ethical aspects of AI, for example the inclusivity and fairness of the information it provides. In this work, we aimed to evaluate different text-to-image AI models and compare the degree of gender bias they present. The evaluated models were Stable Diffusion XL (SDXL), Stable Diffusion Cascade (SC), DALL-E and Emu. We hypothesized that DALL-E and Stable Diffusion, which are comparatively older models, would exhibit a noticeable degree of gender bias towards men, while Emu, which was recently released by Meta AI, would have more balanced results. As hypothesized, we found that both Stable Diffusion models exhibit a noticeable degree of gender bias while Emu demonstrated more balanced results (i.e. less gender bias). However, interestingly, Open AI's DALL-E exhibited almost opposite results, such that the ratio of women to men was significantly higher in most cases tested. Here, although we still observed a bias, the bias favored females over males. This bias may be explained by the fact that OpenAI changed the prompts at its backend, as observed during our experiment. We also observed that Emu from Meta AI utilized user information while generating images via WhatsApp. We also proposed some potential solutions to avoid such biases, including ensuring diversity across AI research teams and having diverse datasets.
Abstract (translated)
随着我们在医疗、金融、电子商务乃至娱乐等行业中越来越多地使用人工智能（AI）进行决策，反思AI的伦理方面变得至关重要，例如它所提供的信息是否具有包容性和公平性。在这项工作中，我们旨在评估不同的文本到图像的AI模型，并比较它们所呈现的性别偏见程度。被评估的模型包括Stable Diffusion XL (SDXL)、Stable Diffusion Cascade (SC)、DALL-E和Emu。我们假设较为老的模型如DALL-E和Stable Diffusion会表现出明显的倾向于男性的性别偏见，而Meta AI最近发布的Emu则会有更加平衡的结果。 正如我们的假设所预测的那样，我们发现两个Stable Diffusion模型确实存在明显的性别偏见，而Emu的表现更为均衡（即性别偏见较少）。然而有趣的是，OpenAI的DALL-E表现出几乎相反的结果，在大多数测试案例中女性与男性的比例明显较高。在这里虽然依然观察到了某种偏差，但这种偏差倾向于有利于女性而非男性。这一现象可能是因为在我们的实验过程中，我们注意到OpenAI在其后台修改了提示词。 此外，我们还发现Meta AI的Emu在通过WhatsApp生成图像时利用了用户信息。基于上述研究结果，我们也提出了一些避免此类偏见的潜在解决方案，包括确保人工智能研究团队的多样性以及使用多样化的数据集。
URL
https://arxiv.org/abs/2509.08004
PDF
-
GS: Generative Segmentation via Label Diffusion
2025-08-27 16:28:15Yuhao Chen, Shubin Chen, Liang Lin, Guangrun WangAbstract
Language-driven image segmentation is a fundamental task in vision-language understanding, requiring models to segment regions of an image corresponding to natural language expressions. Traditional methods approach this as a discriminative problem, assigning each pixel to foreground or background based on semantic alignment. Recently, diffusion models have been introduced to this domain, but existing approaches remain image-centric: they either (i) use image diffusion models as visual feature extractors, (ii) synthesize segmentation data via image generation to train discriminative models, or (iii) perform diffusion inversion to extract attention cues from pre-trained image diffusion models-thereby treating segmentation as an auxiliary process. In this paper, we propose GS (Generative Segmentation), a novel framework that formulates segmentation itself as a generative task via label diffusion. Instead of generating images conditioned on label maps and text, GS reverses the generative process: it directly generates segmentation masks from noise, conditioned on both the input image and the accompanying language description. This paradigm makes label generation the primary modeling target, enabling end-to-end training with explicit control over spatial and semantic fidelity. To demonstrate the effectiveness of our approach, we evaluate GS on Panoptic Narrative Grounding (PNG), a representative and challenging benchmark for multimodal segmentation that requires panoptic-level reasoning guided by narrative captions. Experimental results show that GS significantly outperforms existing discriminative and diffusion-based methods, setting a new state-of-the-art for language-driven segmentation.
Abstract (translated)
语言驱动的图像分割是视觉-语言理解中的基本任务，要求模型根据自然语言表达对图像进行区域划分。传统方法将这个问题视为判别性问题，即基于语义一致性将每个像素分配给前景或背景。最近，扩散模型被引入到这个领域中，但现有的方法仍然以图像为中心：它们要么(i) 使用图像扩散模型作为视觉特征提取器，(ii) 通过图像生成合成分割数据来训练判别模型，或者 (iii) 进行扩散反转，从预训练的图像扩散模型中抽取注意力线索——从而将分割视为辅助过程。在本文中，我们提出了 GS（生成式分割），这是一种新颖的框架，它通过标签扩散将分割本身表述为一个生成性任务。与基于标签图和文本生成图像不同，GS 反转了这一生成过程：它直接从噪声中生成分割掩码，并根据输入图像及其伴随的语言描述进行条件化处理。这种范式使标签生成成为主要的建模目标，使得可以进行端到端训练并显式地控制空间和语义保真度。为了展示我们方法的有效性，我们在 Panoptic Narrative Grounding (PNG) 上评估了 GS，这是一个具有挑战性的多模式分割基准测试，需要通过叙述性标题引导进行全景级别推理。实验结果表明，GS 在语言驱动的分割方面显著优于现有的判别性和基于扩散的方法，并且为这一任务设定了新的最先进水平。
URL
https://arxiv.org/abs/2508.20020
PDF
-
CEIDM: A Controlled Entity and Interaction Diffusion Model for Enhanced Text-to-Image Generation
2025-08-25 07:58:57Mingyue Yang, Dianxi Shi, Jialu Zhou, Xinyu Wei, Leqian Li, Shaowu Yang, Chunping QiuAbstract
In Text-to-Image (T2I) generation, the complexity of entities and their intricate interactions pose a significant challenge for T2I method based on diffusion model: how to effectively control entity and their interactions to produce high-quality images. To address this, we propose CEIDM, a image generation method based on diffusion model with dual controls for entity and interaction. First, we propose an entity interactive relationships mining approach based on Large Language Models (LLMs), extracting reasonable and rich implicit interactive relationships through chain of thought to guide diffusion models to generate high-quality images that are closer to realistic logic and have more reasonable interactive relationships. Furthermore, We propose an interactive action clustering and offset method to cluster and offset the interactive action features contained in each text prompts. By constructing global and local bidirectional offsets, we enhance semantic understanding and detail supplementation of original actions, making the model's understanding of the concept of interactive "actions" more accurate and generating images with more accurate interactive actions. Finally, we design an entity control network which generates masks with entity semantic guidance, then leveraging multi-scale convolutional network to enhance entity feature and dynamic network to fuse feature. It effectively controls entities and significantly improves image quality. Experiments show that the proposed CEIDM method is better than the most representative existing methods in both entity control and their interaction control.
Abstract (translated)
在文本到图像（T2I）生成领域，实体的复杂性和它们之间错综复杂的相互作用对基于扩散模型的方法构成了重大挑战：如何有效控制这些实体及其相互作用以产生高质量的图像。为了解决这一问题，我们提出了CEIDM方法，这是一种基于扩散模型并具备双控机制（用于实体和交互）的图像生成方法。 首先，我们提出了一种基于大型语言模型（LLMs）的实体互动关系挖掘方法，通过链式思考提取合理的、丰富的隐性互动关系，以指导扩散模型生成更接近现实逻辑且具有更多合理互动关系的高质量图像。 其次，为了进一步优化这一过程，我们还提出了一个互动动作聚类和偏移的方法。该方法可以对每个文本提示中包含的互动动作特征进行聚类和偏移处理。通过构建全局和局部双向偏置，这种方法增强了原始行动的语义理解和细节补充能力，使得模型对于“互动行为”的概念理解更加准确，并能生成具有更精确互动行为的图像。 最后，我们设计了一个实体控制网络，该网络在实体语义引导下产生掩码，然后通过多尺度卷积网络增强实体特征并利用动态网络融合这些特征。这一机制有效控制了实体，并显著提升了图像质量。 实验表明，所提出的CEIDM方法不仅在实体控制方面优于现有的最先进方法，在互动控制方面的表现同样出色。
URL
https://arxiv.org/abs/2508.17760
PDF
-
UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding
2025-08-16 07:27:31Yueming Xu, Jiahui Zhang, Ze Huang, Yurui Chen, Yanpeng Zhou, Zhenyu Chen, Yu-Jie Yuan, Pengxiang Xia, Guowei Huang, Xinyue Cai, Zhongang Qi, Xingyue Quan, Jianye Hao, Hang Xu, Li ZhangAbstract
Despite the impressive progress on understanding and generating images shown by the recent unified architectures, the integration of 3D tasks remains challenging and largely unexplored. In this paper, we introduce UniUGG, the first unified understanding and generation framework for 3D modalities. Our unified framework employs an LLM to comprehend and decode sentences and 3D representations. At its core, we propose a spatial decoder leveraging a latent diffusion model to generate high-quality 3D representations. This allows for the generation and imagination of 3D scenes based on a reference image and an arbitrary view transformation, while remaining supports for spatial visual question answering (VQA) tasks. Additionally, we propose a geometric-semantic learning strategy to pretrain the vision encoder. This design jointly captures the input's semantic and geometric cues, enhancing both spatial understanding and generation. Extensive experimental results demonstrate the superiority of our method in visual representation, spatial understanding, and 3D generation. The source code will be released upon paper acceptance.
Abstract (translated)
尽管近期统一架构在理解和生成图像方面取得了令人瞩目的进展，但将三维任务整合进来仍然颇具挑战且尚未被广泛探索。本文介绍了UniUGG，这是首个用于三维模态的统一理解与生成框架。我们的统一框架采用大型语言模型（LLM）来解析和解码句子及3D表示。在核心部分，我们提出了一种基于潜在扩散模型的空间解码器，用以生成高质量的3D表示。这使得根据参考图像以及任意视角转换来生成和想象三维场景成为可能，并且支持空间视觉问答（VQA）任务。此外，我们还提出了几何-语义学习策略来进行视觉编码器的预训练。这一设计同时捕捉输入的语义与几何线索，从而增强空间理解和生成能力。广泛的实验结果证明了我们的方法在视觉表示、空间理解及3D生成方面的优越性。论文一经接受，源代码将公开发布。
URL
https://arxiv.org/abs/2508.11952
PDF
-
Learning User Preferences for Image Generation Model
2025-08-11 17:39:42Wenyi Mo, Ying Ba, Tianyu Zhang, Yalong Bai, Biye LiAbstract
User preference prediction requires a comprehensive and accurate understanding of individual tastes. This includes both surface-level attributes, such as color and style, and deeper content-related aspects, such as themes and composition. However, existing methods typically rely on general human preferences or assume static user profiles, often neglecting individual variability and the dynamic, multifaceted nature of personal taste. To address these limitations, we propose an approach built upon Multimodal Large Language Models, introducing contrastive preference loss and preference tokens to learn personalized user preferences from historical interactions. The contrastive preference loss is designed to effectively distinguish between user ''likes'' and ''dislikes'', while the learnable preference tokens capture shared interest representations among existing users, enabling the model to activate group-specific preferences and enhance consistency across similar users. Extensive experiments demonstrate our model outperforms other methods in preference prediction accuracy, effectively identifying users with similar aesthetic inclinations and providing more precise guidance for generating images that align with individual tastes. The project page is \texttt{this https URL}.
Abstract (translated)
用户偏好预测需要对个人品味进行全面而准确的理解，这包括表层属性（如颜色和风格）以及更深层次的内容相关方面（如主题和构图）。然而，现有的方法通常依赖于一般的人类偏好或假设静态的用户档案，往往忽视了个体差异和个人喜好的动态、多面性。为了解决这些限制，我们提出了一种基于多模态大型语言模型的方法，并引入对比偏好损失和偏好标记来从历史交互中学习个性化的用户偏好。对比偏好损失旨在有效地区分用户的“喜欢”和“不喜欢”，而可学习的偏好标记则捕获现有用户之间的共享兴趣表示，使模型能够激活特定群体的偏好并增强类似用户的一致性。广泛的实验表明，我们的模型在偏好预测准确性方面优于其他方法，可以有效地识别具有相似审美倾向的用户，并为生成与个人品味相符的图像提供更精确的指导。 项目页面位于：\[此URL\]
URL
https://arxiv.org/abs/2508.08220
PDF
-
Diffusion Models with Adaptive Negative Sampling Without External Resources
2025-08-05 00:45:54Alakh Desai, Nuno VasconcelosAbstract
Diffusion models (DMs) have demonstrated an unparalleled ability to create diverse and high-fidelity images from text prompts. However, they are also well-known to vary substantially regarding both prompt adherence and quality. Negative prompting was introduced to improve prompt compliance by specifying what an image must not contain. Previous works have shown the existence of an ideal negative prompt that can maximize the odds of the positive prompt. In this work, we explore relations between negative prompting and classifier-free guidance (CFG) to develop a sampling procedure, {\it Adaptive Negative Sampling Without External Resources} (ANSWER), that accounts for both positive and negative conditions from a single prompt. This leverages the internal understanding of negation by the diffusion model to increase the odds of generating images faithful to the prompt. ANSWER is a training-free technique, applicable to any model that supports CFG, and allows for negative grounding of image concepts without an explicit negative prompts, which are lossy and incomplete. Experiments show that adding ANSWER to existing DMs outperforms the baselines on multiple benchmarks and is preferred by humans 2x more over the other methods.
Abstract (translated)
扩散模型（DM）已经展示了从文本提示生成多样且高质量图像的无与伦比的能力。然而，它们在遵循提示和质量方面也表现出相当大的差异性。为了解决这一问题，引入了负向提示来明确说明图像中不应包含的内容，从而提高对正向提示的遵从度。先前的研究表明存在一个理想的负向提示可以最大化正向提示的成功率。 在这项工作中，我们探讨了负向提示与无分类器指导（CFG）之间的关系，并开发了一种采样程序——“无需外部资源的自适应负样本生成”（ANSWER），该程序能够在单个提示下同时考虑正面和负面条件。此方法利用扩散模型对否定的理解能力来提高图像忠实于提示的概率。ANSWER是一种不需要训练的技术，适用于任何支持CFG的模型，并且可以在没有显式负向提示的情况下实现图像概念的负向锚定，而显式的负向提示往往是低效且不完整的。 实验结果表明，在现有DM中加入ANSWER在多个基准测试上超越了基线方法，并且人类更偏爱该方法2倍于其他方法。
URL
https://arxiv.org/abs/2508.02973
PDF
-
Chain-of-Cooking:Cooking Process Visualization via Bidirectional Chain-of-Thought Guidance
2025-07-29 06:34:59Mengling Xu, Ming Tao, Bing-Kun BaoAbstract
Cooking process visualization is a promising task in the intersection of image generation and food analysis, which aims to generate an image for each cooking step of a recipe. However, most existing works focus on generating images of finished foods based on the given recipes, and face two challenges to visualize the cooking process. First, the appearance of ingredients changes variously across cooking steps, it is difficult to generate the correct appearances of foods that match the textual description, leading to semantic inconsistency. Second, the current step might depend on the operations of previous step, it is crucial to maintain the contextual coherence of images in sequential order. In this work, we present a cooking process visualization model, called Chain-of-Cooking. Specifically, to generate correct appearances of ingredients, we present a Dynamic Patch Selection Module to retrieve previously generated image patches as references, which are most related to current textual contents. Furthermore, to enhance the coherence and keep the rational order of generated images, we propose a Semantic Evolution Module and a Bidirectional Chain-of-Thought (CoT) Guidance. To better utilize the semantics of previous texts, the Semantic Evolution Module establishes the semantical association between latent prompts and current cooking step, and merges it with the latent features. Then the CoT Guidance updates the merged features to guide the current cooking step remain coherent with the previous step. Moreover, we construct a dataset named CookViz, consisting of intermediate image-text pairs for the cooking process. Quantitative and qualitative experiments show that our method outperforms existing methods in generating coherent and semantic consistent cooking process.
Abstract (translated)
烹饪过程可视化是在图像生成和食品分析交叉领域的一项有前景的任务，旨在为食谱中的每一步骤生成一张图像。然而，大多数现有的工作主要集中在根据给定的食谱生成完成食物的图片上，并且在可视化烹饪过程时面临着两个挑战：首先，食材外观会随着步骤的变化而发生变化，因此很难根据文本描述生成匹配的食物外观，从而导致语义不一致；其次，当前步骤可能依赖于前一操作的结果，所以保持图像序列中的上下文连贯性至关重要。为此，在这项工作中我们提出了一种名为“Cooking Chain”的烹饪过程可视化模型。 具体而言，为了生成正确的食材外观，我们设计了一个动态补丁选择模块来检索先前生成的与当前文本内容最相关的图像片段作为参考。此外，为增强图像的一致性和保持合理顺序，我们提出了语义演变模块和双向思考链（CoT）引导策略。为了更好地利用之前文本的语义信息，语义演变模块建立了隐式提示和当前烹饪步骤之间的语义关联，并将其与潜在特征进行融合。随后，CoT 引导更新这些合并的特征以指导当前步骤保持连贯性，使其与前一步骤一致。 此外，我们构建了一个名为CookViz的数据集，该数据集中包含了用于描述整个烹饪过程中的中间图像-文本对。定量和定性的实验表明，在生成连贯且语义一致的烹饪过程中，我们的方法优于现有的其他方法。
URL
https://arxiv.org/abs/2507.21529
PDF
