Abstract
Generating images from prompts containing specific entities requires models to retain as much entity-specific knowledge as possible. However, fully memorizing such knowledge is impractical due to the vast number of entities and their continuous emergence. To address this, we propose Text-based Intelligent Generation with Entity prompt Refinement (TextTIGER), which augments knowledge on entities included in the prompts and then summarizes the augmented descriptions using Large Language Models (LLMs) to mitigate performance degradation from longer inputs. To evaluate our method, we introduce WiT-Cub (WiT with Captions and Uncomplicated Background-explanations), a dataset comprising captions, images, and an entity list. Experiments on four image generation models and five LLMs show that TextTIGER improves image generation performance in standard metrics (IS, FID, and CLIPScore) compared to caption-only prompts. Additionally, multiple annotators' evaluation confirms that the summarized descriptions are more informative, validating LLMs' ability to generate concise yet rich descriptions. These findings demonstrate that refining prompts with augmented and summarized entity-related descriptions enhances image generation capabilities. The code and dataset will be available upon acceptance.
Abstract (translated)
从包含特定实体的提示生成图像要求模型尽可能保留与这些实体相关的知识。然而，由于实体数量庞大且不断涌现，完全记忆这种知识是不切实际的。为解决这一问题，我们提出了基于文本的智能生成与实体提示优化（TextTIGER）方法。该方法增强了提示中包含的实体的知识，并使用大型语言模型（LLMs）对增强后的描述进行总结，以减轻由于输入过长而导致性能下降的问题。 为了评估我们的方法，我们引入了WiT-Cub数据集（带有说明和简洁背景解释的WiT），该数据集包括描述、图像以及一个实体列表。在四种图像生成模型和五种大型语言模型上的实验表明，在标准度量指标（IS、FID 和 CLIPScore）下，TextTIGER相比仅使用描述提示的方法提升了图像生成性能。此外，多位标注者的评估确认了总结后的描述更具信息性，这验证了LLMs能够生成简洁而丰富的描述的能力。 这些发现证明了通过增强和总结实体相关的描述来优化提示可以提升图像生成能力。在论文被接受后，代码和数据集将公开提供。
URL
https://arxiv.org/abs/2504.18269
