Abstract
Generating images with embedded text is crucial for the automatic production of visual and multimodal documents, such as educational materials and advertisements. However, existing diffusion-based text-to-image models often struggle to accurately embed text within images, facing challenges in spelling accuracy, contextual relevance, and visual coherence. Evaluating the ability of such models to embed text within a generated image is complicated due to the lack of comprehensive benchmarks. In this work, we introduce TextInVision, a large-scale, text and prompt complexity driven benchmark designed to evaluate the ability of diffusion models to effectively integrate visual text into images. We crafted a diverse set of prompts and texts that consider various attributes and text characteristics. Additionally, we prepared an image dataset to test Variational Autoencoder (VAE) models across different character representations, highlighting that VAE architectures can also pose challenges in text generation within diffusion frameworks. Through extensive analysis of multiple models, we identify common errors and highlight issues such as spelling inaccuracies and contextual mismatches. By pinpointing the failure points across different prompts and texts, our research lays the foundation for future advancements in AI-generated multimodal content.
Abstract (translated)
将包含嵌入文本的图像生成对于自动生产视觉和多模态文档（如教育材料和广告）至关重要。然而，现有的基于扩散的文字到图像模型在准确地将文字嵌入图像方面经常遇到困难，面临着拼写准确性、上下文相关性和视觉连贯性等方面的挑战。由于缺乏全面的基准测试，评估此类模型在其生成的图像中嵌入文本的能力变得复杂。在这项工作中，我们引入了TextInVision，这是一个大规模的、以文本和提示复杂度驱动的基准测试系统，旨在评估扩散模型将视觉文字有效整合到图像中的能力。我们精心设计了一组多样化的提示语和文本，考虑了各种属性和文本特征。此外，我们还准备了一个图像数据集来测试变分自编码器（VAE）模型在不同字符表示上的性能，并强调VAE架构同样可能面临在扩散框架内生成文字方面的挑战。通过对多个模型进行广泛的分析，我们识别出常见的错误并指出了诸如拼写不准确和上下文不符等问题。通过确定不同类型提示语和文本中的失败点，我们的研究为未来AI生成的多模态内容的进步奠定了基础。
URL
https://arxiv.org/abs/2503.13730
