Scene_Text
-
Smart Routing for Multimodal Video Retrieval: When to Search What
2025-07-12 15:45:03Kevin Dela RosaAbstract
We introduce ModaRoute, an LLM-based intelligent routing system that dynamically selects optimal modalities for multimodal video retrieval. While dense text captions can achieve 75.9% Recall@5, they require expensive offline processing and miss critical visual information present in 34% of clips with scene text not captured by ASR. By analyzing query intent and predicting information needs, ModaRoute reduces computational overhead by 41% while achieving 60.9% Recall@5. Our approach uses GPT-4.1 to route queries across ASR (speech), OCR (text), and visual indices, averaging 1.78 modalities per query versus exhaustive 3.0 modality search. Evaluation on 1.8M video clips demonstrates that intelligent routing provides a practical solution for scaling multimodal retrieval systems, reducing infrastructure costs while maintaining competitive effectiveness for real-world deployment.
Abstract (translated)
我们介绍了一种基于大型语言模型（LLM）的智能路由系统——ModaRoute，该系统能够动态选择最佳模式来进行多模态视频检索。尽管密集的文字字幕可以达到75.9%的召回率@5指标，但它们需要昂贵的离线处理，并且会错过34%包含场景文本的片段中的关键视觉信息（这些信息未被自动语音识别（ASR）捕捉到）。通过分析查询意图并预测信息需求，ModaRoute在减少计算开销41%的同时实现了60.9%的召回率@5。我们的方法使用GPT-4.1将查询路由到ASR（语音）、OCR（文本）和视觉索引之间，在每个查询中平均使用了1.78种模式，而全面搜索则需要3.0种模式。在对180万视频片段进行的评估显示，智能路由为扩展多模态检索系统提供了实际解决方案，能够降低基础设施成本并保持现实世界部署中的竞争力效果。
URL
https://arxiv.org/abs/2507.13374
PDF
-
ESTR-CoT: Towards Explainable and Accurate Event Stream based Scene Text Recognition with Chain-of-Thought Reasoning
2025-07-02 23:41:31Xiao Wang, Jingtao Jiang, Qiang Chen, Lan Chen, Lin Zhu, Yaowei Wang, Yonghong Tian, Jin TangAbstract
Event stream based scene text recognition is a newly arising research topic in recent years which performs better than the widely used RGB cameras in extremely challenging scenarios, especially the low illumination, fast motion. Existing works either adopt end-to-end encoder-decoder framework or large language models for enhanced recognition, however, they are still limited by the challenges of insufficient interpretability and weak contextual logical reasoning. In this work, we propose a novel chain-of-thought reasoning based event stream scene text recognition framework, termed ESTR-CoT. Specifically, we first adopt the vision encoder EVA-CLIP (ViT-G/14) to transform the input event stream into tokens and utilize a Llama tokenizer to encode the given generation prompt. A Q-former is used to align the vision token to the pre-trained large language model Vicuna-7B and output both the answer and chain-of-thought (CoT) reasoning process simultaneously. Our framework can be optimized using supervised fine-tuning in an end-to-end manner. In addition, we also propose a large-scale CoT dataset to train our framework via a three stage processing (i.e., generation, polish, and expert verification). This dataset provides a solid data foundation for the development of subsequent reasoning-based large models. Extensive experiments on three event stream STR benchmark datasets (i.e., EventSTR, WordArt*, IC15*) fully validated the effectiveness and interpretability of our proposed framework. The source code and pre-trained models will be released on this https URL.
Abstract (translated)
基于事件流的场景文本识别是近年来新兴的研究课题，该技术在极端挑战性环境下（尤其是低光照和快速移动）的表现优于广泛使用的RGB相机。现有研究工作要么采用端到端编码器-解码器框架，要么利用大型语言模型来增强识别性能，但它们仍然受限于解释性和弱情境逻辑推理的不足。 在此项工作中，我们提出了一种基于链式思维（chain-of-thought, CoT）推理的事件流场景文本识别框架，命名为ESTR-CoT。具体来说，首先采用视觉编码器EVA-CLIP (ViT-G/14) 将输入的事件流转换为标记，并利用Llama分词器对给定的生成提示进行编码。Q-former用于将视觉标记与预训练的大规模语言模型Vicuna-7B对齐，同时输出答案和链式思维推理过程。我们的框架可以通过监督微调以端到端的方式进行优化。 此外，我们还提出了一套大规模CoT数据集，通过三阶段处理（生成、润色及专家验证）来训练我们的框架。该数据集为后续基于推理的大型模型开发提供了坚实的数据基础。 在三个事件流STR基准数据集（即EventSTR、WordArt*、IC15*）上的广泛实验充分证实了我们所提框架的有效性和可解释性。源代码和预训练模型将在以下网址发布：[https URL]。
URL
https://arxiv.org/abs/2507.02200
PDF
-
WordCon: Word-level Typography Control in Scene Text Rendering
2025-06-26 14:00:38Wenda Shi, Yiren Song, Zihan Rao, Dengming Zhang, Jiaming Liu, Xingxing ZouAbstract
Achieving precise word-level typography control within generated images remains a persistent challenge. To address it, we newly construct a word-level controlled scene text dataset and introduce the Text-Image Alignment (TIA) framework. This framework leverages cross-modal correspondence between text and local image regions provided by grounding models to enhance the Text-to-Image (T2I) model training. Furthermore, we propose WordCon, a hybrid parameter-efficient fine-tuning (PEFT) method. WordCon reparameterizes selective key parameters, improving both efficiency and portability. This allows seamless integration into diverse pipelines, including artistic text rendering, text editing, and image-conditioned text rendering. To further enhance controllability, the masked loss at the latent level is applied to guide the model to concentrate on learning the text region in the image, and the joint-attention loss provides feature-level supervision to promote disentanglement between different words. Both qualitative and quantitative results demonstrate the superiority of our method to the state of the art. The datasets and source code will be available for academic use.
Abstract (translated)
在生成图像中实现精确的单词级排版控制仍然是一个持续性的挑战。为了解决这个问题，我们构建了一个新的以单词级别进行控制的场景文本数据集，并引入了文本-图像对齐（TIA）框架。该框架利用了由定位模型提供的文本与局部图像区域之间的跨模态对应关系，以此来增强从文本到图像（Text-to-Image, T2I）模型的训练效果。此外，我们还提出了WordCon，这是一种混合参数高效的微调（PEFT）方法。WordCon通过对选择的关键参数进行重新参数化，从而在提高效率和便携性的同时改进了这些关键参数的表现。这使得它可以无缝地整合到各种不同的工作流程中，包括艺术文字渲染、文本编辑以及基于图像的文字渲染等方面。 为了进一步增强控制能力，我们在潜在层面上应用了掩码损失（masked loss），以引导模型专注于学习图像中的文字区域，并且联合注意力损失（joint-attention loss）提供了特征级监督来促进不同单词之间的分离。无论是定性还是定量的结果都证明了我们的方法在现有技术中处于领先地位。 我们将提供数据集和源代码供学术使用。
URL
https://arxiv.org/abs/2506.21276
PDF
-
Inverse Scene Text Removal
2025-06-26 04:32:35Takumi Yoshimatsu, Shumpei Takezaki, Seiichi UchidaAbstract
Scene text removal (STR) aims to erase textual elements from images. It was originally intended for removing privacy-sensitiveor undesired texts from natural scene images, but is now also appliedto typographic images. STR typically detects text regions and theninpaints them. Although STR has advanced through neural networksand synthetic data, misuse risks have increased. This paper investi-gates Inverse STR (ISTR), which analyzes STR-processed images andfocuses on binary classification (detecting whether an image has un-dergone STR) and localizing removed text regions. We demonstrate inexperiments that these tasks are achievable with high accuracies, en-abling detection of potential misuse and improving STR. We also at-tempt to recover the removed text content by training a text recognizerto understand its difficulty.
Abstract (translated)
场景文本去除（STR）的目标是从图像中移除文字元素。最初，它被设计用来从自然场景图片中删除隐私敏感或不希望存在的文字，但现在也应用于排版图象。STR通常会检测出文字区域然后进行修复。尽管STR通过神经网络和合成数据得到了改进，但滥用的风险也随之增加。本文研究了逆向STR（ISTR），即分析经过STR处理的图像，并专注于二元分类（识别一张图片是否经历过STR处理）以及定位被移除的文字区域。我们在实验中展示了这些任务可以实现高准确率，从而能够检测潜在的滥用行为并改进STR技术。我们还尝试通过训练一个文字识别器来理解恢复删除文本内容的难度。
URL
https://arxiv.org/abs/2506.21002
PDF
-
MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling
2025-06-12 11:54:13Liang Yin, Xudong Xie, Zhang Li, Xiang Bai, Yuliang LiuAbstract
Scene text retrieval has made significant progress with the assistance of accurate text localization. However, existing approaches typically require costly bounding box annotations for training. Besides, they mostly adopt a customized retrieval strategy but struggle to unify various types of queries to meet diverse retrieval needs. To address these issues, we introduce Muti-query Scene Text retrieval with Attention Recycling (MSTAR), a box-free approach for scene text retrieval. It incorporates progressive vision embedding to dynamically capture the multi-grained representation of texts and harmonizes free-style text queries with style-aware instructions. Additionally, a multi-instance matching module is integrated to enhance vision-language alignment. Furthermore, we build the Multi-Query Text Retrieval (MQTR) dataset, the first benchmark designed to evaluate the multi-query scene text retrieval capability of models, comprising four query types and 16k images. Extensive experiments demonstrate the superiority of our method across seven public datasets and the MQTR dataset. Notably, MSTAR marginally surpasses the previous state-of-the-art model by 6.4% in MAP on Total-Text while eliminating box annotation costs. Moreover, on the MQTR benchmark, MSTAR significantly outperforms the previous models by an average of 8.5%. The code and datasets are available at this https URL.
Abstract (translated)
场景文本检索在精确的文字定位技术的帮助下取得了显著进步。然而，现有的方法通常需要昂贵的边界框标注来进行训练。此外，这些方法大多采用定制化的检索策略，但在统一各种类型的查询以满足多样化的检索需求方面存在困难。为了解决这些问题，我们引入了带有注意力循环（Attention Recycling）的多查询场景文本检索(MSTAR)，这是一种无需边界的场景文本检索方法。MSTAR采用了渐进式视觉嵌入技术，能够动态捕捉文本的多层次表示，并将自由风格的文字查询与风格感知指令和谐统一。此外，还集成了一个多实例匹配模块以增强视觉语言对齐。 为了评估模型在多查询场景文本检索能力上的表现，我们构建了多查询文本检索(MQTR)数据集，这是首个专门为此任务设计的基准测试平台，包含四种类型的查询和16,000张图像。广泛的实验表明，在七个公开的数据集和MQTR数据集上，我们的方法均表现出优越性。值得注意的是，MSTAR在Total-Text数据集中以MAP（平均精度）为指标超越了前最先进模型6.4%，同时消除了边界框标注的成本。此外，在MQTR基准测试中，MSTAR的性能显著优于之前的模型，平均高出8.5%。 代码和数据集可以在以下链接获取：[此URL]（请将此处的"this https URL"替换为实际链接）。
URL
https://arxiv.org/abs/2506.10609
PDF
-
When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding
2025-06-05 19:53:19Yan Shu, Hangui Lin, Yexin Liu, Yan Zhang, Gangyan Zeng, Yan Li, Yu Zhou, Ser-Nam Lim, Harry Yang, Nicu SebeAbstract
Large Multimodal Models (LMMs) have achieved impressive progress in visual perception and reasoning. However, when confronted with visually ambiguous or non-semantic scene text, they often struggle to accurately spot and understand the content, frequently generating semantically plausible yet visually incorrect answers, which we refer to as semantic hallucination. In this work, we investigate the underlying causes of semantic hallucination and identify a key finding: Transformer layers in LLM with stronger attention focus on scene text regions are less prone to producing semantic hallucinations. Thus, we propose a training-free semantic hallucination mitigation framework comprising two key components: (1) ZoomText, a coarse-to-fine strategy that identifies potential text regions without external detectors; and (2) Grounded Layer Correction, which adaptively leverages the internal representations from layers less prone to hallucination to guide decoding, correcting hallucinated outputs for non-semantic samples while preserving the semantics of meaningful ones. To enable rigorous evaluation, we introduce TextHalu-Bench, a benchmark of over 1,730 samples spanning both semantic and non-semantic cases, with manually curated question-answer pairs designed to probe model hallucinations. Extensive experiments demonstrate that our method not only effectively mitigates semantic hallucination but also achieves strong performance on public benchmarks for scene text spotting and understanding.
Abstract (translated)
大型多模态模型（LMMs）在视觉感知和推理方面取得了显著进展。然而，面对视觉模糊或非语义场景文本时，它们往往难以准确识别和理解内容，并常常生成语义上看似合理但实际上与图像不符的答案，我们称之为语义幻觉。在这项工作中，我们探讨了导致语义幻觉的潜在原因并发现了一个关键点：大型语言模型（LLM）中注意力更强的Transformer层在处理场景文本区域时较少出现语义幻觉。基于这一发现，我们提出了一种无训练方法来减轻语义幻觉框架，该框架包含两个核心组件： 1. **ZoomText**：一种从粗到细策略，无需外部检测器即可识别潜在的文字区域。 2. **Grounded Layer Correction**（地面层校正）：此模块会根据模型中较不易产生幻觉的层数内部表示来调整解码过程，在不损害有意义样本语义的情况下纠正非语义样本的输出。 为了严格评估该方法的效果，我们引入了TextHalu-Bench，这是一个包含超过1,730个样本的基准测试集，这些样本涵盖了从有语义到无语义的各种场景，并配有人工策划的问题-答案对，旨在探测模型可能出现的幻觉。广泛的实验表明，我们的方法不仅有效减轻了语义幻觉问题，在公共场景文本检测和理解基准上也取得了很强的表现。
URL
https://arxiv.org/abs/2506.05551
PDF
-
Beyond Cropped Regions: New Benchmark and Corresponding Baseline for Chinese Scene Text Retrieval in Diverse Layouts
Abstract
Chinese scene text retrieval is a practical task that aims to search for images containing visual instances of a Chinese query text. This task is extremely challenging because Chinese text often features complex and diverse layouts in real-world scenes. Current efforts tend to inherit the solution for English scene text retrieval, failing to achieve satisfactory performance. In this paper, we establish a Diversified Layout benchmark for Chinese Street View Text Retrieval (DL-CSVTR), which is specifically designed to evaluate retrieval performance across various text layouts, including vertical, cross-line, and partial alignments. To address the limitations in existing methods, we propose Chinese Scene Text Retrieval CLIP (CSTR-CLIP), a novel model that integrates global visual information with multi-granularity alignment training. CSTR-CLIP applies a two-stage training process to overcome previous limitations, such as the exclusion of visual features outside the text region and reliance on single-granularity alignment, thereby enabling the model to effectively handle diverse text layouts. Experiments on existing benchmark show that CSTR-CLIP outperforms the previous state-of-the-art model by 18.82% accuracy and also provides faster inference speed. Further analysis on DL-CSVTR confirms the superior performance of CSTR-CLIP in handling various text layouts. The dataset and code will be publicly available to facilitate research in Chinese scene text retrieval.
Abstract (translated)
中文场景文本检索是一项实用的任务，旨在搜索包含查询文本视觉实例的图像。由于实际场景中中文文本经常具有复杂多样的布局形式，这一任务极具挑战性。目前的研究通常沿用了英文场景文本检索的方法，导致无法取得令人满意的性能。 在本文中，我们建立了一个多样化布局基准（Diversified Layout benchmark）用于评估中文街景文本检索（DL-CSVTR），该基准特别设计来评估不同文本布局下的检索性能，包括竖直、跨行和部分对齐的场景。为了解决现有方法中的局限性，我们提出了一个新的模型——中文场景文本检索CLIP (CSTR-CLIP)，这个模型整合了全局视觉信息并采用多粒度对齐训练策略。 CSTR-CLIP 通过两阶段训练过程克服了先前模型的一些限制，比如忽略文字区域外的视觉特征和依赖单一粒度的对齐方式。这使得模型能够有效地处理多样化的文本布局。 在现有基准上的实验表明，与最先进的方法相比，CSTR-CLIP 在准确率上提高了18.82%，并且提供了更快的推理速度。进一步在 DL-CSVTR 上的分析证实了 CSTR-CLIP 处理各种文本布局时的优越性。数据集和代码将公开发布，以促进中文场景文本检索的研究进展。 通过这种方式，CSTR-CLIP 不仅提升了模型在处理复杂、多变中文街景文本任务上的表现，同时也为研究人员提供了更加全面且实用的数据资源与工具支持。
URL
https://arxiv.org/abs/2506.04999
PDF
-
TextVidBench: A Benchmark for Long Video Scene Text Understanding
2025-06-05 12:54:56Yangyang Zhong, Ji Qi, Yuan Yao, Pengxin Luo, Yunfeng Yan, Donglian Qi, Zhiyuan Liu, Tat-Seng ChuaAbstract
Despite recent progress on the short-video Text-Visual Question Answering (ViteVQA) task - largely driven by benchmarks such as M4-ViteVQA - existing datasets still suffer from limited video duration and narrow evaluation scopes, making it difficult to adequately assess the growing capabilities of powerful multimodal large language models (MLLMs). To address these limitations, we introduce TextVidBench, the first benchmark specifically designed for long-video text question answering (>3 minutes). TextVidBench makes three key contributions: 1) Cross-domain long-video coverage: Spanning 9 categories (e.g., news, sports, gaming), with an average video length of 2306 seconds, enabling more realistic evaluation of long-video understanding. 2) A three-stage evaluation framework: "Text Needle-in-Haystack -> Temporal Grounding -> Text Dynamics Captioning". 3) High-quality fine-grained annotations: Containing over 5,000 question-answer pairs with detailed semantic labeling. Furthermore, we propose an efficient paradigm for improving large models through: (i) introducing the IT-Rope mechanism and temporal prompt engineering to enhance temporal perception, (ii) adopting non-uniform positional encoding to better handle long video sequences, and (iii) applying lightweight fine-tuning on video-text data. Extensive experiments on multiple public datasets as well as TextVidBench demonstrate that our new benchmark presents significant challenges to existing models, while our proposed method offers valuable insights into improving long-video scene text understanding capabilities.
Abstract (translated)
尽管在短视频文本-视觉问答（ViteVQA）任务上取得了近期进展——这主要得益于M4-ViteVQA等基准测试的推动，现有的数据集仍然存在视频时长有限和评估范围狭窄的问题，使得难以充分评价强大多模态大型语言模型（MLLMs）的能力。为了解决这些问题，我们引入了TextVidBench，这是第一个专门针对长时间视频文本问答（>3分钟）设计的基准。 TextVidBench做出了三项关键贡献： 1. **跨域长视频覆盖**：涵盖9个类别（例如新闻、体育、游戏），平均视频长度为2306秒，这使得对长视频理解的评估更加现实。 2. **三阶段评估框架**：“文本大海捞针 -> 时间定位 -> 文本动态描述”。 3. **高质量细粒度注释**：包含超过5,000个问题-答案配对，并带有详细的语义标签。 此外，我们还提出了一种通过以下方式改进大规模模型的高效范式： (i) 引入IT-Rope机制和时间提示工程来增强时间感知能力； (ii) 采用非均匀位置编码以更好地处理长视频序列； (iii) 在视频文本数据上进行轻量级微调。 在多个公共数据集以及TextVidBench上的广泛实验表明，我们的新基准对现有模型提出了重大挑战，而我们提出的改进方法为提升长时间视频场景中的文本理解能力提供了有价值的见解。
URL
https://arxiv.org/abs/2506.04983
PDF
-
TextSR: Diffusion Super-Resolution with Multilingual OCR Guidance
2025-05-29 05:40:35Keren Ye, Ignacio Garcia Dorado, Michalis Raptis, Mauricio Delbracio, Irene Zhu, Peyman Milanfar, Hossein TalebiAbstract
While recent advancements in Image Super-Resolution (SR) using diffusion models have shown promise in improving overall image quality, their application to scene text images has revealed limitations. These models often struggle with accurate text region localization and fail to effectively model image and multilingual character-to-shape priors. This leads to inconsistencies, the generation of hallucinated textures, and a decrease in the perceived quality of the super-resolved text. To address these issues, we introduce TextSR, a multimodal diffusion model specifically designed for Multilingual Scene Text Image Super-Resolution. TextSR leverages a text detector to pinpoint text regions within an image and then employs Optical Character Recognition (OCR) to extract multilingual text from these areas. The extracted text characters are then transformed into visual shapes using a UTF-8 based text encoder and cross-attention. Recognizing that OCR may sometimes produce inaccurate results in real-world scenarios, we have developed two innovative methods to enhance the robustness of our model. By integrating text character priors with the low-resolution text images, our model effectively guides the super-resolution process, enhancing fine details within the text and improving overall legibility. The superior performance of our model on both the TextZoom and TextVQA datasets sets a new benchmark for STISR, underscoring the efficacy of our approach.
Abstract (translated)
尽管最近使用扩散模型在图像超分辨率（SR）方面取得了进展，这些方法在提高整体图像质量方面显示出潜力，但它们在应用于场景文本图像时却暴露出了一些局限性。这些模型往往难以准确地定位文本区域，并且无法有效建模图像和多语言字符到形状的先验知识，这导致了不一致、生成幻觉纹理以及超分辨率后文本感知质量下降的问题。 为了应对这些问题，我们引入了一种专为多语言场景文本图像超分辨率设计的TextSR多模式扩散模型。TextSR利用一个文本检测器来识别图像中的文本区域，并使用光学字符识别（OCR）从这些区域中提取多语言文本。然后，通过基于UTF-8的文本编码和跨注意力机制将提取出的文字转换为视觉形状。 考虑到在实际场景中，OCR可能会产生不准确的结果，我们开发了两种创新的方法来增强模型的鲁棒性。通过结合低分辨率图像中的字符先验知识，我们的模型能够有效地指导超分辨率过程，提高文字细节并提升整体可读性。我们在TextZoom和TextVQA数据集上的测试结果表明，我们的方法在场景文本图像超分辨率（STISR）领域树立了新的性能标杆，这进一步证明了我们方法的有效性和优越性。
URL
https://arxiv.org/abs/2505.23119
PDF
-
Syn3DTxt: Embedding 3D Cues for Scene Text Generation
2025-05-24 02:53:24Li-Syun Hsiung, Jun-Kai Tu, Kuan-Wu Chu, Yu-Hsuan Chiu, Yan-Tsung Peng, Sheng-Luen Chung, Gee-Sern Jison HsuAbstract
This study aims to investigate the challenge of insufficient three-dimensional context in synthetic datasets for scene text rendering. Although recent advances in diffusion models and related techniques have improved certain aspects of scene text generation, most existing approaches continue to rely on 2D data, sourcing authentic training examples from movie posters and book covers, which limits their ability to capture the complex interactions among spatial layout and visual effects in real-world scenes. In particular, traditional 2D datasets do not provide the necessary geometric cues for accurately embedding text into diverse backgrounds. To address this limitation, we propose a novel standard for constructing synthetic datasets that incorporates surface normals to enrich three-dimensional scene characteristic. By adding surface normals to conventional 2D data, our approach aims to enhance the representation of spatial relationships and provide a more robust foundation for future scene text rendering methods. Extensive experiments demonstrate that datasets built under this new standard offer improved geometric context, facilitating further advancements in text rendering under complex 3D-spatial conditions.
Abstract (translated)
这项研究旨在探讨合成数据集中三维上下文不足对场景文本渲染的挑战。尽管最近在扩散模型及相关技术方面的进展改善了场景文本生成的一些方面，但大多数现有方法仍然依赖于二维数据，并从电影海报和书籍封面等来源获取真实的训练示例，这限制了它们捕捉现实世界场景中空间布局与视觉效果之间复杂互动的能力。特别是，传统的二维数据集无法提供将文本准确嵌入到各种背景所需的几何线索。 为了解决这一局限性，我们提出了一种新的构建合成数据集的标准，该标准引入表面法线以丰富三维场景特性。通过向常规的二维数据添加表面法线，我们的方法旨在增强空间关系的表现，并为未来的场景文本渲染方法提供更坚实的基石。广泛的实验表明，在这种新标准下建立的数据集提供了改进的几何上下文，有助于在复杂的三维空间条件下进一步推进文本渲染技术的进步。
URL
https://arxiv.org/abs/2505.18479
PDF
-
TextFlux: An OCR-Free DiT Model for High-Fidelity Multilingual Scene Text Synthesis
2025-05-23 11:46:46Yu Xie, Jielei Zhang, Pengyu Chen, Ziyue Wang, Weihang Wang, Longwen Gao, Peiyi Li, Huyang Sun, Qiang Zhang, Qian Qiao, Jiaqing Fan, Zhouhui LianAbstract
Diffusion-based scene text synthesis has progressed rapidly, yet existing methods commonly rely on additional visual conditioning modules and require large-scale annotated data to support multilingual generation. In this work, we revisit the necessity of complex auxiliary modules and further explore an approach that simultaneously ensures glyph accuracy and achieves high-fidelity scene integration, by leveraging diffusion models' inherent capabilities for contextual reasoning. To this end, we introduce TextFlux, a DiT-based framework that enables multilingual scene text synthesis. The advantages of TextFlux can be summarized as follows: (1) OCR-free model architecture. TextFlux eliminates the need for OCR encoders (additional visual conditioning modules) that are specifically used to extract visual text-related features. (2) Strong multilingual scalability. TextFlux is effective in low-resource multilingual settings, and achieves strong performance in newly added languages with fewer than 1,000 samples. (3) Streamlined training setup. TextFlux is trained with only 1% of the training data required by competing methods. (4) Controllable multi-line text generation. TextFlux offers flexible multi-line synthesis with precise line-level control, outperforming methods restricted to single-line or rigid layouts. Extensive experiments and visualizations demonstrate that TextFlux outperforms previous methods in both qualitative and quantitative evaluations.
Abstract (translated)
基于扩散模型的场景文本合成技术取得了快速进展，但现有方法通常依赖额外的视觉条件模块，并且需要大规模标注数据来支持多语言生成。在这项工作中，我们重新审视了复杂辅助模块的必要性，并进一步探索了一种同时确保字形准确性并实现高质量场景整合的方法，通过利用扩散模型在上下文推理方面的内在能力。为此，我们介绍了基于DiT框架的TextFlux系统，它能够进行多语言场景文本合成。TextFlux的优势可以总结如下： 1. **无OCR架构**：TextFlux消除了专门用于提取视觉文本相关特征的OCR编码器（额外的视觉条件模块）的需求。 2. **强大的多语言可扩展性**：在低资源环境下，TextFlux仍然非常有效，并且在新添加的语言中使用少于1,000个样本也能实现强劲性能。 3. **简化的训练设置**：与竞争方法相比，TextFlux仅需1%的训练数据即可完成训练。 4. **可控多行文本生成**：TextFlux提供灵活的多行合成，并具有精确到每行级别的控制能力，超越了限制于单行或固定布局的方法。 大量的实验和可视化结果证明，在定性和定量评估中，TextFlux均优于先前的方法。
URL
https://arxiv.org/abs/2505.17778
PDF
-
The Devil is in Fine-tuning and Long-tailed Problems:A New Benchmark for Scene Text Detection
2025-05-21 15:26:46Tianjiao Cao, Jiahao Lyu, Weichao Zeng, Weimin Mu, Yu ZhouAbstract
Scene text detection has seen the emergence of high-performing methods that excel on academic benchmarks. However, these detectors often fail to replicate such success in real-world scenarios. We uncover two key factors contributing to this discrepancy through extensive experiments. First, a \textit{Fine-tuning Gap}, where models leverage \textit{Dataset-Specific Optimization} (DSO) paradigm for one domain at the cost of reduced effectiveness in others, leads to inflated performances on academic benchmarks. Second, the suboptimal performance in practical settings is primarily attributed to the long-tailed distribution of texts, where detectors struggle with rare and complex categories as artistic or overlapped text. Given that the DSO paradigm might undermine the generalization ability of models, we advocate for a \textit{Joint-Dataset Learning} (JDL) protocol to alleviate the Fine-tuning Gap. Additionally, an error analysis is conducted to identify three major categories and 13 subcategories of challenges in long-tailed scene text, upon which we propose a Long-Tailed Benchmark (LTB). LTB facilitates a comprehensive evaluation of ability to handle a diverse range of long-tailed challenges. We further introduce MAEDet, a self-supervised learning-based method, as a strong baseline for LTB. The code is available at this https URL.
Abstract (translated)
场景文本检测领域已涌现出一系列在学术基准测试中表现出色的方法，然而这些方法往往难以在实际环境中复制同样的成功。通过广泛的实验研究，我们发现了导致这一差距的两个关键因素。首先，存在一个称为“微调间隙”（Fine-tuning Gap）的现象：模型利用特定数据集优化（Dataset-Specific Optimization, DSO）范式为某一领域量身定制时，会导致在其他领域的有效性降低，并且这种做法会使得模型在学术基准测试中的表现被夸大。其次，在实际应用中性能不佳主要是由于文本的长尾分布导致的，即检测器难以处理罕见和复杂的类别，如艺术性或重叠的文字。 鉴于DSO范式可能削弱模型的泛化能力，我们提倡采用联合数据集学习（Joint-Dataset Learning, JDL）协议来缓解微调间隙问题。此外，我们还进行了一项错误分析，识别出长尾场景文本中的三个主要挑战类别和十三个子类别，并提出一个长尾基准测试（Long-Tailed Benchmark, LTB），以全面评估处理多样化的长尾挑战的能力。 为了进一步支持这一研究，我们引入了MAEDet方法作为LTB的一个强基线，该方法基于自监督学习。相关的代码可以在以下链接中获取：[此URL]（请将实际的GitHub或其他代码托管平台的URL填入括号内）。
URL
https://arxiv.org/abs/2505.15649
PDF
-
GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing
2025-05-08 03:11:58Tong Wang, Ting Liu, Xiaochao Qu, Chengjing Wu, Luoqi Liu, Xiaolin HuAbstract
Scene text editing, a subfield of image editing, requires modifying texts in images while preserving style consistency and visual coherence with the surrounding environment. While diffusion-based methods have shown promise in text generation, they still struggle to produce high-quality results. These methods often generate distorted or unrecognizable characters, particularly when dealing with complex characters like Chinese. In such systems, characters are composed of intricate stroke patterns and spatial relationships that must be precisely maintained. We present GlyphMastero, a specialized glyph encoder designed to guide the latent diffusion model for generating texts with stroke-level precision. Our key insight is that existing methods, despite using pretrained OCR models for feature extraction, fail to capture the hierarchical nature of text structures - from individual strokes to stroke-level interactions to overall character-level structure. To address this, our glyph encoder explicitly models and captures the cross-level interactions between local-level individual characters and global-level text lines through our novel glyph attention module. Meanwhile, our model implements a feature pyramid network to fuse the multi-scale OCR backbone features at the global-level. Through these cross-level and multi-scale fusions, we obtain more detailed glyph-aware guidance, enabling precise control over the scene text generation process. Our method achieves an 18.02\% improvement in sentence accuracy over the state-of-the-art multi-lingual scene text editing baseline, while simultaneously reducing the text-region Fréchet inception distance by 53.28\%.
Abstract (translated)
场景文本编辑是图像编辑的一个子领域，它要求在保留风格一致性和周围环境视觉连贯性的前提下修改图片中的文字。虽然基于扩散的方法在文本生成方面显示出潜力，但它们仍然难以产生高质量的结果。这些方法经常生成扭曲或无法辨认的字符，尤其是在处理像中文这样复杂的字符时。在这种系统中，字符由精细的笔画模式和空间关系构成，必须精确维护。 我们提出了GlyphMastero，这是一种专门设计用于引导潜在扩散模型以实现笔画级别精度的文字生成技术的字体编码器。我们的关键见解是现有的方法虽然使用了预训练的OCR（光学字符识别）模型进行特征提取，但未能捕捉文本结构的层次性——从单个笔画到笔画级别的交互再到整个字符级别的结构。为了解决这个问题，我们的字体编码器通过我们新颖的字体注意模块明确地建模并捕获局部级别个体字符与全局级别文字行之间的跨层级互动。 同时，我们的模型实现了一个特征金字塔网络来融合多尺度OCR骨干特性在全局层面的信息。通过这些跨层级和多层次的融合，我们获得了更为详细的文字感知指导，从而能够对场景文本生成过程进行更精确的控制。 我们的方法相对于最先进的多语言场景文本编辑基线，在句子准确度方面提高了18.02%，同时将文本区域Fréchet inception距离减少了53.28%。
URL
https://arxiv.org/abs/2505.04915
PDF
-
FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editing
2025-05-06 08:56:28Rui Lan, Yancheng Bai, Xu Duan, Mingxing Li, Lei Sun, Xiangxiang ChuAbstract
The task of scene text editing is to modify or add texts on images while maintaining the fidelity of newly generated text and visual coherence with the background. Recent works based on latent diffusion models (LDM) show improved text editing results, yet still face challenges and often generate inaccurate or unrecognizable characters, especially for non-Latin ones (\eg, Chinese), which have complex glyph structures. To address these issues, we present FLUX-Text, a simple and advanced multilingual scene text editing framework based on FLUX-Fill. Specifically, we carefully investigate glyph conditioning, considering both visual and textual modalities. To retain the original generative capabilities of FLUX-Fill while enhancing its understanding and generation of glyphs, we propose lightweight glyph and text embedding modules. Owning to the lightweight design, FLUX-Text is trained only with $100K$ training examples compared to current popular methods trained with 2.9M ones. With no bells and whistles, our method achieves state-of-the-art performance on text editing tasks. Qualitative and quantitative experiments on the public datasets demonstrate that our method surpasses previous works in text fidelity.
Abstract (translated)
场景文本编辑的任务是在保持新生成的文字质量和与背景图像视觉一致性的前提下，修改或增加图片中的文字。最近基于潜在扩散模型（LDM）的工作提高了文本编辑的效果，但仍然面临挑战，并且常常产生不准确或难以辨认的字符，尤其是对于非拉丁文（如中文），这些文字具有复杂的字形结构。为了应对这些问题，我们提出了FLUX-Text，这是一个基于FLUX-Fill的简单而先进的多语言场景文本编辑框架。具体来说，我们仔细研究了字形条件化，同时考虑视觉和文本模态。 为了保留FLUX-Fill原有的生成能力并增强其对字形的理解与生成，我们提出了一种轻量级的字形和文本嵌入模块。由于采用了轻量化设计，FLUX-Text仅使用10万个训练样本进行训练，而目前流行的方法则需要290万个训练样本。在没有任何复杂技巧的情况下，我们的方法在文本编辑任务上达到了最先进的性能水平。通过公共数据集上的定性和定量实验表明，我们的方法在文字保真度方面超越了之前的工作。
URL
https://arxiv.org/abs/2505.03329
PDF
-
Analysing the Robustness of Vision-Language-Models to Common Corruptions
2025-04-18 13:46:32Muhammad Usama, Syeda Aisha Asim, Syed Bilal Ali, Syed Talal Wasim, Umair Bin MansoorAbstract
Vision-language models (VLMs) have demonstrated impressive capabilities in understanding and reasoning about visual and textual content. However, their robustness to common image corruptions remains under-explored. In this work, we present the first comprehensive analysis of VLM robustness across 19 corruption types from the ImageNet-C benchmark, spanning four categories: noise, blur, weather, and digital distortions. We introduce two new benchmarks, TextVQA-C and GQA-C, to systematically evaluate how corruptions affect scene text understanding and object-based reasoning, respectively. Our analysis reveals that transformer-based VLMs exhibit distinct vulnerability patterns across tasks: text recognition deteriorates most severely under blur and snow corruptions, while object reasoning shows higher sensitivity to corruptions such as frost and impulse noise. We connect these observations to the frequency-domain characteristics of different corruptions, revealing how transformers' inherent bias toward low-frequency processing explains their differential robustness patterns. Our findings provide valuable insights for developing more corruption-robust vision-language models for real-world applications.
Abstract (translated)
视觉语言模型（VLMs）在理解和推理视觉和文本内容方面表现出令人印象深刻的能力。然而，它们对常见图像损坏的鲁棒性仍鲜有研究。在这项工作中，我们首次全面分析了来自ImageNet-C基准测试的19种不同类型图像损坏对VLM鲁棒性的影响，这些类型涵盖了四个类别：噪声、模糊、天气和数字扭曲。我们介绍了两个新的基准测试——TextVQA-C和GQA-C，分别系统地评估图像损坏如何影响场景文本理解和基于对象的推理。我们的分析揭示了基于变压器的VLM在不同任务中表现出独特的脆弱性模式：文本识别在遭受模糊和雪破坏时退化最为严重；而物体推理则对霜冻和脉冲噪声等破坏表现出更高的敏感度。我们将这些观察结果与不同类型损坏的频域特性联系起来，展示了变压器模型固有的低频处理偏见如何解释其不同的鲁棒性模式。我们的发现为开发更适用于现实世界应用、能更好地抵抗图像损坏影响的视觉语言模型提供了宝贵的见解。
URL
https://arxiv.org/abs/2504.13690
PDF
-
TSAL: Few-shot Text Segmentation Based on Attribute Learning
2025-04-15 13:12:42Chenming Li, Chengxu Liu, Yuanting Fan, Xiao Jin, Xingsong Hou, Xueming QianAbstract
Recently supervised learning rapidly develops in scene text segmentation. However, the lack of high-quality datasets and the high cost of pixel annotation greatly limit the development of them. Considering the well-performed few-shot learning methods for downstream tasks, we investigate the application of the few-shot learning method to scene text segmentation. We propose TSAL, which leverages CLIP's prior knowledge to learn text attributes for segmentation. To fully utilize the semantic and texture information in the image, a visual-guided branch is proposed to separately extract text and background features. To reduce data dependency and improve text detection accuracy, the adaptive prompt-guided branch employs effective adaptive prompt templates to capture various text attributes. To enable adaptive prompts capture distinctive text features and complex background distribution, we propose Adaptive Feature Alignment module(AFA). By aligning learnable tokens of different attributes with visual features and prompt prototypes, AFA enables adaptive prompts to capture both general and distinctive attribute information. TSAL can capture the unique attributes of text and achieve precise segmentation using only few images. Experiments demonstrate that our method achieves SOTA performance on multiple text segmentation datasets under few-shot settings and show great potential in text-related domains.
Abstract (translated)
最近，监督学习在场景文本分割领域取得了迅速发展。然而，高质量数据集的缺乏和像素标注成本高昂极大地限制了该领域的进步。鉴于针对下游任务表现良好的少量样本学习方法，我们研究了将少量样本学习方法应用于场景文本分割的应用。为此，我们提出了TSAL（Text Segmentation with Adaptive Prompt Learning），它利用CLIP的先验知识来学习用于分割的文字属性。为了充分利用图像中的语义和纹理信息，提出了一种视觉引导分支，以分别提取文字和背景特征。 为减少对数据的依赖并提高文本检测精度，适应性提示引导分支采用了有效的自适应提示模板来捕捉各种文字属性。为了让自适应提示能够捕获独特且复杂的文字特征及背景分布，我们提出了自适应特征对齐模块（Adaptive Feature Alignment, AFA）。通过将不同属性的学习令牌与视觉特征和提示原型进行对齐，AFA使自适应提示能够同时捕捉通用和独特的属性信息。 TSAL能够在仅使用少量图像的情况下捕获文本的独特属性，并实现精确的分割。实验表明，在多个文本分割数据集下的少量样本设置中，我们的方法达到了最先进的性能，并在与文本相关的领域展示了巨大的潜力。
URL
https://arxiv.org/abs/2504.11164
PDF
-
SemiETS: Integrating Spatial and Content Consistencies for Semi-Supervised End-to-end Text Spotting
2025-04-14 08:09:17Dongliang Luo, Hanshen Zhu, Ziyang Zhang, Dingkang Liang, Xudong Xie, Yuliang Liu, Xiang BaiAbstract
Most previous scene text spotting methods rely on high-quality manual annotations to achieve promising performance. To reduce their expensive costs, we study semi-supervised text spotting (SSTS) to exploit useful information from unlabeled images. However, directly applying existing semi-supervised methods of general scenes to SSTS will face new challenges: 1) inconsistent pseudo labels between detection and recognition tasks, and 2) sub-optimal supervisions caused by inconsistency between teacher/student. Thus, we propose a new Semi-supervised framework for End-to-end Text Spotting, namely SemiETS that leverages the complementarity of text detection and recognition. Specifically, it gradually generates reliable hierarchical pseudo labels for each task, thereby reducing noisy labels. Meanwhile, it extracts important information in locations and transcriptions from bidirectional flows to improve consistency. Extensive experiments on three datasets under various settings demonstrate the effectiveness of SemiETS on arbitrary-shaped text. For example, it outperforms previous state-of-the-art SSL methods by a large margin on end-to-end spotting (+8.7%, +5.6%, and +2.6% H-mean under 0.5%, 1%, and 2% labeled data settings on Total-Text, respectively). More importantly, it still improves upon a strongly supervised text spotter trained with plenty of labeled data by 2.0%. Compelling domain adaptation ability shows practical potential. Moreover, our method demonstrates consistent improvement on different text spotters.
Abstract (translated)
大多数以前的场景文本识别方法依赖于高质量的手工标注来实现较好的性能。为了减少昂贵的成本，我们研究了半监督文本定位（SSTS），以利用未标记图像中的有用信息。然而，直接将现有的通用场景半监督方法应用于SSTS会面临新的挑战：1）检测和识别任务之间的伪标签不一致；2）由于教师/学生之间的一致性问题导致的次优监督信号。因此，我们提出了一种新的用于端到端文本定位的半监督框架——SemiETS，该框架利用了文本检测与识别间的互补关系。具体来说，它逐步为每个任务生成可靠的分层伪标签，从而减少噪声标签的数量。同时，它从双向流动中提取位置和转录中的重要信息以提高一致性。在三个不同设置的数据集上的广泛实验表明，SemiETS在任意形状文本的定位上表现出有效性。例如，在Total-Text数据集中，对于端到端检测任务，其分别在0.5%，1% 和2% 标注数据的条件下超过了之前的最先进SSL方法8.7%，5.6%，和2.6%H-means。更重要的是，它仍然比经过大量标注数据训练的强大监督文本定位器提高了2.0% 的性能。强大的领域适应能力显示了其实用潜力。此外，我们的方法在不同的文本定位器上均表现出一致的改进效果。
URL
https://arxiv.org/abs/2504.09966
PDF
-
Edge Approximation Text Detector
2025-04-05 00:12:51Chuang Yang, Xu Han, Tao Han, Han Han, Bingxuan Zhao, Qi WangAbstract
Pursuing efficient text shape representations helps scene text detection models focus on compact foreground regions and optimize the contour reconstruction steps to simplify the whole detection pipeline. Current approaches either represent irregular shapes via box-to-polygon strategy or decomposing a contour into pieces for fitting gradually, the deficiency of coarse contours or complex pipelines always exists in these models. Considering the above issues, we introduce EdgeText to fit text contours compactly while alleviating excessive contour rebuilding processes. Concretely, it is observed that the two long edges of texts can be regarded as smooth curves. It allows us to build contours via continuous and smooth edges that cover text regions tightly instead of fitting piecewise, which helps avoid the two limitations in current models. Inspired by this observation, EdgeText formulates the text representation as the edge approximation problem via parameterized curve fitting functions. In the inference stage, our model starts with locating text centers, and then creating curve functions for approximating text edges relying on the points. Meanwhile, truncation points are determined based on the location features. In the end, extracting curve segments from curve functions by using the pixel coordinate information brought by truncation points to reconstruct text contours. Furthermore, considering the deep dependency of EdgeText on text edges, a bilateral enhanced perception (BEP) module is designed. It encourages our model to pay attention to the recognition of edge features. Additionally, to accelerate the learning of the curve function parameters, we introduce a proportional integral loss (PI-loss) to force the proposed model to focus on the curve distribution and avoid being disturbed by text scales.
Abstract (translated)
追求高效的文本形状表示有助于场景文本检测模型聚焦于紧致的前景区域，并优化轮廓重建步骤以简化整个检测流程。当前的方法要么通过盒到多边形策略来表示不规则形状，要么将一个轮廓分解成多个片段逐步拟合。这些方法中往往存在粗略轮廓或复杂流程的问题。 为了解决上述问题，我们提出了EdgeText模型，旨在紧凑地匹配文本轮廓同时缓解过度的轮廓重建过程。具体而言，观察到文本的两条长边可以视为平滑曲线，这使得我们可以构建连续且平滑覆盖文本区域的边缘以代替分段拟合的方式，从而避免现有模型中的两个局限性。受到这一发现启发，EdgeText将文本表示形式化为参数化曲线拟合函数的问题。 在推理阶段，我们的模型首先定位文字中心，然后基于这些点创建用于近似文字边界的曲线函数。同时，根据位置特征确定截断点。最后，利用由截断点带来的像素坐标信息从曲线函数中提取曲线段来重建文本轮廓。 此外，考虑到EdgeText对文本边缘的深度依赖性，我们设计了一种双边增强感知（BEP）模块，以鼓励模型注意边缘特性的识别。为了加速学习曲线函数参数的速度，我们引入了比例积分损失（PI-loss），使提出的模型专注于曲线分布并避免受到文字尺度的影响。
URL
https://arxiv.org/abs/2504.04001
PDF
-
NCAP: Scene Text Image Super-Resolution with Non-CAtegorical Prior
2025-04-01 04:14:07Dongwoo Park, Suk Pil KoAbstract
Scene text image super-resolution (STISR) enhances the resolution and quality of low-resolution images. Unlike previous studies that treated scene text images as natural images, recent methods using a text prior (TP), extracted from a pre-trained text recognizer, have shown strong performance. However, two major issues emerge: (1) Explicit categorical priors, like TP, can negatively impact STISR if incorrect. We reveal that these explicit priors are unstable and propose replacing them with Non-CAtegorical Prior (NCAP) using penultimate layer representations. (2) Pre-trained recognizers used to generate TP struggle with low-resolution images. To address this, most studies jointly train the recognizer with the STISR network to bridge the domain gap between low- and high-resolution images, but this can cause an overconfidence phenomenon in the prior modality. We highlight this issue and propose a method to mitigate it by mixing hard and soft labels. Experiments on the TextZoom dataset demonstrate an improvement by 3.5%, while our method significantly enhances generalization performance by 14.8\% across four text recognition datasets. Our method generalizes to all TP-guided STISR networks.
Abstract (translated)
场景文本图像超分辨率（STISR）技术旨在提高低分辨率图像的分辨率和质量。与以往将场景文本图像视为自然图像处理的研究不同，最近使用从预训练的文字识别器中提取的文本先验（TP）的方法展现了强大的性能。然而，这种方法存在两个主要问题： 1. 显式的类别先验，如TP，在错误的情况下会负面影响STISR的效果。我们发现这些显式先验不稳定，并提出用来自倒数第二层表示的非分类前缀（NCAP）来替代它们。 2. 用于生成TP的预训练识别器在处理低分辨率图像时遇到困难。为了应对这一问题，大多数研究通过将识别器与STISR网络联合训练来弥合低分辨率和高分辨率图像之间的领域差距，但这可能会导致先验模态中的过度自信现象。我们强调了这个问题，并提出了一种混合硬标签和软标签的方法来缓解它。 在TextZoom数据集上的实验表明，我们的方法相比基准提高了3.5%的性能，而在四个文本识别数据集中，我们的方法显著增强了泛化性能，增幅达14.8%。此外，我们提出的方法能够应用于所有基于TP引导的STISR网络中。
URL
https://arxiv.org/abs/2504.00410
PDF
-
Beyond Words: Advancing Long-Text Image Generation via Multimodal Autoregressive Models
2025-03-26 03:44:25Alex Jinpeng Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, Min LiAbstract
Recent advancements in autoregressive and diffusion models have led to strong performance in image generation with short scene text words. However, generating coherent, long-form text in images, such as paragraphs in slides or documents, remains a major challenge for current generative models. We present the first work specifically focused on long text image generation, addressing a critical gap in existing text-to-image systems that typically handle only brief phrases or single sentences. Through comprehensive analysis of state-of-the-art autoregressive generation models, we identify the image tokenizer as a critical bottleneck in text generating quality. To address this, we introduce a novel text-focused, binary tokenizer optimized for capturing detailed scene text features. Leveraging our tokenizer, we develop \ModelName, a multimodal autoregressive model that excels in generating high-quality long-text images with unprecedented fidelity. Our model offers robust controllability, enabling customization of text properties such as font style, size, color, and alignment. Extensive experiments demonstrate that \ModelName~significantly outperforms SD3.5 Large~\cite{sd3} and GPT4o~\cite{gpt4o} with DALL-E 3~\cite{dalle3} in generating long text accurately, consistently, and flexibly. Beyond its technical achievements, \ModelName~opens up exciting opportunities for innovative applications like interleaved document and PowerPoint generation, establishing a new frontier in long-text image generating.
Abstract (translated)
最近在自回归和扩散模型方面取得的进展已经使得使用简短场景文本词生成图像时表现出色。然而，当前的生成模型仍然难以克服一个重大挑战：即在图像中生成连贯且长篇幅的文字，例如幻灯片或文档中的段落。我们首次提出了专注于长文本图像生成的工作，填补了现有文字到图像系统的一个关键空白，这些系统通常只能处理简短语句或单个句子。 通过对最先进的自回归生成模型进行全面分析，我们发现图像标记器是影响文本生成质量的关键瓶颈。为了解决这一问题，我们引入了一种新颖的以文本为中心、二进制化的标记器，该标记器专为捕捉详细的场景文字特征而优化。利用我们的标记器，我们开发了\ModelName（假设这里的\ModelName是指一个具体的模型名称），这是一种多模态自回归模型，在生成高质量且细节丰富的长篇幅文字图像方面表现卓越，并以前所未有的精确度实现了这一目标。 该模型提供强大的可控制性，能够根据需求定制文本属性，如字体样式、大小、颜色和对齐方式。广泛的实验表明，\ModelName在准确、一致和灵活地生成长文本方面显著优于SD3.5 Large~\cite{sd3}、GPT4o~\cite{gpt4o}与DALL-E 3~\cite{dalle3}的组合。 除了技术成就之外，\ModelName还为创新应用开辟了令人兴奋的机会，例如交错生成文档和PowerPoint演示文稿，从而在长篇幅文字图像生成领域开创了一个新的前沿。
URL
https://arxiv.org/abs/2503.20198
PDF
