Abstract
Generative modeling is widely regarded as one of the most essential problems in today's AI community, with text-to-image generation having gained unprecedented real-world impacts. Among various approaches, diffusion models have achieved remarkable success and have become the de facto solution for text-to-image generation. However, despite their impressive performance, these models exhibit fundamental limitations in adhering to numerical constraints in user instructions, frequently generating images with an incorrect number of objects. While several prior works have mentioned this issue, a comprehensive and rigorous evaluation of this limitation remains lacking. To address this gap, we introduce T2ICountBench, a novel benchmark designed to rigorously evaluate the counting ability of state-of-the-art text-to-image diffusion models. Our benchmark encompasses a diverse set of generative models, including both open-source and private systems. It explicitly isolates counting performance from other capabilities, provides structured difficulty levels, and incorporates human evaluations to ensure high reliability. Extensive evaluations with T2ICountBench reveal that all state-of-the-art diffusion models fail to generate the correct number of objects, with accuracy dropping significantly as the number of objects increases. Additionally, an exploratory study on prompt refinement demonstrates that such simple interventions generally do not improve counting accuracy. Our findings highlight the inherent challenges in numerical understanding within diffusion models and point to promising directions for future improvements.
Abstract (translated)
生成式模型被广泛认为是当今AI社区中最基本的问题之一，其中文本到图像的生成已经产生了前所未有的现实世界影响。在各种方法中，扩散模型取得了显著的成功，并已成为文本到图像生成的标准解决方案。然而，尽管它们表现出色，这些模型在遵守用户指令中的数值约束方面存在根本性限制，经常生成包含错误对象数量的图片。虽然一些先前的研究工作提到了这个问题，但对其进行全面而严格的评估仍然不足。 为了填补这一空白，我们引入了T2ICountBench，这是一个新型基准测试平台，旨在严格评估最先进的文本到图像扩散模型在计数方面的能力。该基准涵盖了各种生成式模型，包括开源和专有系统，并明确地将计数性能与其他能力区分开来。它提供结构化的难度级别，并结合了人类评价以确保高可靠性。 使用T2ICountBench进行的广泛评估表明，所有最先进的扩散模型在生成正确的对象数量方面都失败了，随着对象数量的增加，准确性显著下降。此外，关于提示调整的研究发现，此类简单干预通常不会提高计数精度。 我们的研究结果突出了扩散模型中数值理解所面临的内在挑战，并为未来改进指明了有前景的方向。
URL
https://arxiv.org/abs/2503.06884
