Abstract
Text-to-Image models, including Stable Diffusion, have significantly improved in generating images that are highly semantically aligned with the given prompts. However, existing models may fail to produce appropriate images for the cultural concepts or objects that are not well known or underrepresented in western cultures, such as `hangari' (Korean utensil). In this paper, we propose a novel approach, Culturally-Aware Text-to-Image Generation with Iterative Prompt Refinement (Culture-TRIP), which refines the prompt in order to improve the alignment of the image with such culture nouns in text-to-image models. Our approach (1) retrieves cultural contexts and visual details related to the culture nouns in the prompt and (2) iteratively refines and evaluates the prompt based on a set of cultural criteria and large language models. The refinement process utilizes the information retrieved from Wikipedia and the Web. Our user survey, conducted with 66 participants from eight different countries demonstrates that our proposed approach enhances the alignment between the images and the prompts. In particular, C-TRIP demonstrates improved alignment between the generated images and underrepresented culture nouns. Resource can be found at this https URL.
Abstract (translated)
文本到图像模型（如Stable Diffusion）在生成与给定提示高度语义对齐的图像方面取得了显著进步。然而，现有的模型可能无法为西方文化中不常见或代表性不足的文化概念或物品（例如韩国器皿“hangari”）生成适当的图像。为此，在本文中我们提出了一种新颖的方法——具有迭代提示优化的文化感知文本到图像生成（Culture-TRIP），该方法通过改进提示来提高这些文化名词在文本到图像模型中的对齐度。 我们的方法包括： 1. 检索与提示中文化名词相关的文化背景和视觉细节； 2. 根据一系列文化和大规模语言模型的评估，迭代优化和评价提示。此过程利用从维基百科和网络检索的信息。 我们进行了一项用户调查，共66名来自8个不同国家的参与者参与其中。调查显示我们的方法能够增强图像与提示之间的对齐度，特别是在生成图片与其文化名词的相关性方面得到了改进。相关资源可在提供的链接中找到。
URL
https://arxiv.org/abs/2502.16902
