Abstract
Text-to-image (T2I) models have garnered significant attention for generating high-quality images aligned with text prompts. However, rapid T2I model advancements reveal limitations in early benchmarks, lacking comprehensive evaluations, for example, the evaluation on reasoning, text rendering and style. Notably, recent state-of-the-art models, with their rich knowledge modeling capabilities, show promising results on the image generation problems requiring strong reasoning ability, yet existing evaluation systems have not adequately addressed this frontier. To systematically address these gaps, we introduce OneIG-Bench, a meticulously designed comprehensive benchmark framework for fine-grained evaluation of T2I models across multiple dimensions, including prompt-image alignment, text rendering precision, reasoning-generated content, stylization, and diversity. By structuring the evaluation, this benchmark enables in-depth analysis of model performance, helping researchers and practitioners pinpoint strengths and bottlenecks in the full pipeline of image generation. Specifically, OneIG-Bench enables flexible evaluation by allowing users to focus on a particular evaluation subset. Instead of generating images for the entire set of prompts, users can generate images only for the prompts associated with the selected dimension and complete the corresponding evaluation accordingly. Our codebase and dataset are now publicly available to facilitate reproducible evaluation studies and cross-model comparisons within the T2I research community.
Abstract (translated)
文本到图像（T2I）模型因其能生成高质量并与文本提示相匹配的图像而备受关注。然而，随着T2I模型的快速进步，早期基准测试中出现了一些局限性，例如缺乏全面评估，特别是在推理、文本渲染和风格方面。值得注意的是，最近最先进的模型在需要强大推理能力的图像生成问题上展现了令人鼓舞的结果，但现有的评估系统尚未充分解决这一领域的问题。为了系统地填补这些空白，我们引入了OneIG-Bench，这是一个精心设计的基准框架，旨在对T2I模型进行细致全面的多维度评估，包括提示-图像匹配度、文本渲染精度、推理生成内容、风格化和多样性等方面。通过结构化的评估方法，该基准能够深入分析模型性能，帮助研究人员和实践者识别整个图像生成流程中的优势与瓶颈。具体而言，OneIG-Bench允许用户灵活地进行特定评估子集的测试，而非为所有提示生成图像，使用者只需针对选定维度的相关提示生成图像，并完成相应的评估即可。我们的代码库和数据集现已公开发布，便于T2I研究社区内的可重复性研究及跨模型比较。
URL
https://arxiv.org/abs/2506.07977
