Abstract
The advancements in the domain of LLMs in recent years have surprised many, showcasing their remarkable capabilities and diverse applications. Their potential applications in various real-world scenarios have led to significant research on their reliability and effectiveness. On the other hand, multimodal LLMs and Text-to-Image models have only recently gained prominence, especially when compared to text-only LLMs. Their reliability remains constrained due to insufficient research on assessing their performance and robustness. This paper aims to establish a comprehensive evaluation framework for Text-to-Image models, concentrating particularly on their adherence to prompts. We created a novel dataset that aimed to assess the robustness of these models in generating images that conform to the specified factors of variation in the input text prompts. Our evaluation studies present findings on three variants of Stable Diffusion models: Stable Diffusion 3 Medium, Stable Diffusion 3.5 Large, and Stable Diffusion 3.5 Large Turbo, and two variants of Janus models: Janus Pro 1B and Janus Pro 7B. We introduce a pipeline that leverages text descriptions generated by the gpt-4o model for our ground-truth images, which are then used to generate artificial images by passing these descriptions to the Text-to-Image models. We then pass these generated images again through gpt-4o using the same system prompt and compare the variation between the two descriptions. Our results reveal that these models struggle to create simple binary images with only two factors of variation: a simple geometric shape and its location. We also show, using pre-trained VAEs on our dataset, that they fail to generate images that follow our input dataset distribution.
Abstract (translated)
近年来，大型语言模型（LLM）领域取得了许多令人惊讶的进步，展示了它们在多种场景中的显著能力和广泛应用。由于这些模型在实际应用场景中显示出巨大的潜力，因此对其可靠性和有效性进行了大量的研究。相比之下，多模态的LLM和文本到图像模型直到最近才开始受到关注，特别是在与纯文本LLM相比时。尽管如此，由于缺乏对它们性能和鲁棒性的评估研究，其可靠性仍然受限。 本文旨在为文本到图像（Text-to-Image）模型建立一个全面的评价框架，特别关注这些模型在生成符合输入文本提示中指定因素变化的图像方面的表现能力。为此，我们创建了一个新的数据集，用于评估这些模型在上述条件下的稳健性。我们的研究对三种Stable Diffusion模型进行了评测：Stable Diffusion 3 Medium、Stable Diffusion 3.5 Large以及Stable Diffusion 3.5 Large Turbo；同时还有两种Janus模型：Janus Pro 1B和Janus Pro 7B。 为了进行这项评估，我们引入了一个管道系统，该系统利用gpt-4o生成的文本描述作为真实图像的基础，并将这些描述传递给Text-to-Image模型以生成人工图像。然后，我们将生成的图像再次通过相同的系统提示传递给gpt-4o，并比较两个描述之间的变化。 我们的结果表明，在只有两种因素（一个简单的几何形状及其位置）发生二元变化的情况下，这些模型在创建简单二元图象方面遇到困难。此外，我们还使用预训练的变分自编码器（VAE），并在我们的数据集上展示了它们无法生成符合输入数据分布特性的图像。 总的来说，这项研究揭示了现有Text-to-Image模型的一些局限性，并为进一步改善这些模型提供了有价值的见解和建议。
URL
https://arxiv.org/abs/2507.08039
