Scene_Text
-
CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy
2024-12-03 07:03:25Zhibo Yang, Jun Tang, Zhaohai Li, Pengfei Wang, Jianqiang Wan, Humen Zhong, Xuejing Liu, Mingkun Yang, Peng Wang, Yuliang Liu, LianWen Jin, Xiang Bai, Shuai Bai, Junyang LinAbstract
Large Multimodal Models (LMMs) have demonstrated impressive performance on recognizing document images with natural language instructions. However, it remains unclear to what extent capabilities in literacy with rich structure and fine-grained visual challenges. The current landscape lacks a comprehensive benchmark to effectively measure the literate capabilities of LMMs. Existing benchmarks are often limited by narrow scenarios and specified tasks. To this end, we introduce CC-OCR, a comprehensive benchmark that possess a diverse range of scenarios, tasks, and challenges. CC-OCR comprises four OCR-centric tracks: multi-scene text reading, multilingual text reading, document parsing, and key information extraction. It includes 39 subsets with 7,058 full annotated images, of which 41% are sourced from real applications, being released for the first time. Furthermore, we evaluate nine prominent LMMs and reveal both the strengths and weaknesses of these models, particularly in text grounding, multi-orientation, and hallucination of repetition. CC-OCR aims to comprehensively evaluate the capabilities of LMMs on OCR-centered tasks, driving advancement in LMMs.
Abstract (translated)
URL
https://arxiv.org/abs/2412.02210
PDF
-
TextSSR: Diffusion-based Data Synthesis for Scene Text Recognition
2024-12-02 05:26:25Xingsong Ye, Yongkun Du, Yunbo Tao, Zhineng ChenAbstract
Scene text recognition (STR) suffers from the challenges of either less realistic synthetic training data or the difficulty of collecting sufficient high-quality real-world data, limiting the effectiveness of trained STR models. Meanwhile, despite producing holistically appealing text images, diffusion-based text image generation methods struggle to generate accurate and realistic instance-level text on a large scale. To tackle this, we introduce TextSSR: a novel framework for Synthesizing Scene Text Recognition data via a diffusion-based universal text region synthesis model. It ensures accuracy by focusing on generating text within a specified image region and leveraging rich glyph and position information to create the less complex text region compared to the entire image. Furthermore, we utilize neighboring text within the region as a prompt to capture real-world font styles and layout patterns, guiding the generated text to resemble actual scenes. Finally, due to its prompt-free nature and capability for character-level synthesis, TextSSR enjoys a wonderful scalability and we construct an anagram-based TextSSR-F dataset with 0.4 million text instances with complexity and realism. Experiments show that models trained on added TextSSR-F data exhibit better accuracy compared to models trained on 4 million existing synthetic data. Moreover, its accuracy margin to models trained fully on a real-world dataset is less than 3.7%, confirming TextSSR's effectiveness and its great potential in scene text image synthesis. Our code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2412.01137
PDF
-
SVTRv2: CTC Beats Encoder-Decoder Models in Scene Text Recognition
2024-11-24 14:21:35Yongkun Du, Zhineng Chen, Hongtao Xie, Caiyan Jia, Yu-Gang JiangAbstract
Connectionist temporal classification (CTC)-based scene text recognition (STR) methods, e.g., SVTR, are widely employed in OCR applications, mainly due to their simple architecture, which only contains a visual model and a CTC-aligned linear classifier, and therefore fast inference. However, they generally have worse accuracy than encoder-decoder-based methods (EDTRs), particularly in challenging scenarios. In this paper, we propose SVTRv2, a CTC model that beats leading EDTRs in both accuracy and inference speed. SVTRv2 introduces novel upgrades to handle text irregularity and utilize linguistic context, which endows it with the capability to deal with challenging and diverse text instances. First, a multi-size resizing (MSR) strategy is proposed to adaptively resize the text and maintain its readability. Meanwhile, we introduce a feature rearrangement module (FRM) to ensure that visual features accommodate the alignment requirement of CTC well, thus alleviating the alignment puzzle. Second, we propose a semantic guidance module (SGM). It integrates linguistic context into the visual model, allowing it to leverage language information for improved accuracy. Moreover, SGM can be omitted at the inference stage and would not increase the inference cost. We evaluate SVTRv2 in both standard and recent challenging benchmarks, where SVTRv2 is fairly compared with 24 mainstream STR models across multiple scenarios, including different types of text irregularity, languages, and long text. The results indicate that SVTRv2 surpasses all the EDTRs across the scenarios in terms of accuracy and speed. Code is available at this https URL.
Abstract (translated)
基于连接主义时间分类（CTC）的场景文本识别（STR）方法，例如SVTR，在OCR应用中被广泛使用，主要归因于其简单的架构，仅包含一个视觉模型和一个与CTC对齐的线性分类器，因此推理速度较快。然而，它们通常比基于编解码器的方法（EDTRs）具有更低的准确性，尤其是在挑战性场景下表现不佳。本文中，我们提出了SVTRv2，这是一种在准确性和推理速度上都超越领先EDTRs的CTC模型。SVTRv2引入了新颖的功能升级来处理文本不规则性和利用语言上下文，使其具备应对复杂多样的文字实例的能力。首先，提出了一种多尺寸调整（MSR）策略以自适应地调整文本并保持其可读性。同时，我们引入了一个特征重新排列模块（FRM），确保视觉特性能够很好地满足CTC的对齐要求，从而缓解了对齐难题。其次，我们提出了一个语义引导模块（SGM）。它将语言上下文整合到视觉模型中，允许利用语言信息提高准确性。此外，在推理阶段可以省略SGM，不会增加推理成本。我们在标准和近期具有挑战性的基准测试上评估了SVTRv2，与跨多种场景的24种主流STR模型进行了公平比较，包括不同类型的文本不规则性、语言以及长文本。结果表明，SVTRv2在准确性和速度方面超越了所有EDTRs。代码可在以下网址获取：此 https URL。
URL
https://arxiv.org/abs/2411.15858
PDF
-
Boosting Semi-Supervised Scene Text Recognition via Viewing and Summarizing
2024-11-23 15:24:47Yadong Qu, Yuxin Wang, Bangbang Zhou, Zixiao Wang, Hongtao Xie, Yongdong ZhangAbstract
Existing scene text recognition (STR) methods struggle to recognize challenging texts, especially for artistic and severely distorted characters. The limitation lies in the insufficient exploration of character morphologies, including the monotonousness of widely used synthetic training data and the sensitivity of the model to character morphologies. To address these issues, inspired by the human learning process of viewing and summarizing, we facilitate the contrastive learning-based STR framework in a self-motivated manner by leveraging synthetic and real unlabeled data without any human cost. In the viewing process, to compensate for the simplicity of synthetic data and enrich character morphology diversity, we propose an Online Generation Strategy to generate background-free samples with diverse character styles. By excluding background noise distractions, the model is encouraged to focus on character morphology and generalize the ability to recognize complex samples when trained with only simple synthetic data. To boost the summarizing process, we theoretically demonstrate the derivation error in the previous character contrastive loss, which mistakenly causes the sparsity in the intra-class distribution and exacerbates ambiguity on challenging samples. Therefore, a new Character Unidirectional Alignment Loss is proposed to correct this error and unify the representation of the same characters in all samples by aligning the character features in the student model with the reference features in the teacher model. Extensive experiment results show that our method achieves SOTA performance (94.7\% and 70.9\% average accuracy on common benchmarks and Union14M-Benchmark). Code will be available at this https URL.
Abstract (translated)
现有的场景文本识别（STR）方法在识别具有挑战性的文本方面存在困难，特别是对于艺术性和严重扭曲的字符。其局限性在于对字符形态探索不足，包括广泛使用的合成训练数据单调以及模型对字符形态敏感。为了解决这些问题，受人类学习过程中的观察和总结启发，我们通过利用合成和现实未标记的数据，在无需人工成本的情况下，以自我激励的方式促进对比学习为基础的STR框架的发展。在观察过程中，为了弥补合成数据的简单性并丰富字符形态多样性，我们提出了一种在线生成策略来生成具有多样化字符风格且无背景的样本。通过排除背景噪音干扰，模型被鼓励专注于字符形态，并能够仅使用简单的合成数据进行训练时泛化识别复杂样本的能力。为了增强总结过程，我们在理论上证明了先前字符对比损失中的推导错误，该错误误导致类内分布稀疏并加剧了对具有挑战性样本的模糊性。因此，我们提出了一个新的字符单向对齐损失来纠正这一错误并通过在学生模型中将字符特征与教师模型中的参考特征对齐统一所有样本中相同字符的表现形式。广泛的实验结果表明，我们的方法实现了最先进的性能（在常见基准和Union14M-Benchmark上的平均准确率分别为94.7%和70.9%）。代码将在以下链接提供：此 https URL。
URL
https://arxiv.org/abs/2411.15585
PDF
-
Relational Contrastive Learning and Masked Image Modeling for Scene Text Recognition
2024-11-18 01:11:47Tiancheng Lin, Jinglei Zhang, Yi Xu, Kai Chen, Rui Zhang, Chang-Wen ChenAbstract
Context-aware methods have achieved remarkable advancements in supervised scene text recognition by leveraging semantic priors from words. Considering the heterogeneity of text and background in STR, we propose that such contextual priors can be reinterpreted as the relations between textual elements, serving as effective self-supervised labels for representation learning. However, textual relations are restricted to the finite size of the dataset due to lexical dependencies, which causes over-fitting problem, thus compromising the representation quality. To address this, our work introduces a unified framework of Relational Contrastive Learning and Masked Image Modeling for STR (RCMSTR), which explicitly models the enriched textual relations. For the RCL branch, we first introduce the relational rearrangement module to cultivate new relations on the fly. Based on this, we further conduct relational contrastive learning to model the intra- and inter-hierarchical relations for frames, sub-words and this http URL the other hand, MIM can naturally boost the context information via masking, where we find that the block masking strategy is more effective for STR. For the effective integration of RCL and MIM, we also introduce a novel decoupling design aimed at mitigating the impact of masked images on contrastive learning. Additionally, to enhance the compatibility of MIM with CNNs, we propose the adoption of sparse convolutions and directly sharing the weights with dense convolutions in training. The proposed RCMSTR demonstrates superior performance in various evaluation protocols for different STR-related downstream tasks, outperforming the existing state-of-the-art self-supervised STR techniques. Ablation studies and qualitative experimental results further validate the effectiveness of our this http URL code and pre-trained models will be available at this https URL .
Abstract (translated)
上下文感知的方法通过利用单词中的语义先验，在监督场景文本识别中取得了显著的进步。考虑到STR中文本和背景的异质性，我们提出可以将这些上下文先验重新解释为文本元素之间的关系，作为表示学习的有效自监督标签。然而，由于词汇依赖性，文本关系受到数据集有限大小的限制，这会导致过拟合问题，从而影响表示的质量。为了应对这一挑战，我们的工作提出了一个用于STR（RCMSTR）的关系对比学习和掩码图像建模统一框架，该框架明确地建模了丰富的文本关系。 对于RCL分支，我们首先引入了一个关系重新排列模块来实时培养新的关系。基于此，我们进一步进行关系对比学习以模型帧、子词之间以及与之相关的层级关系。另一方面，MIM可以通过掩码自然地增强上下文信息，在STR中发现块掩码策略更为有效。为了实现RCL和MIM的有效集成，我们也引入了一种新颖的解耦设计，旨在减轻被掩码图像对对比学习的影响。此外，为了提高MIM与CNNs之间的兼容性，我们建议采用稀疏卷积，并直接在训练过程中与密集卷积共享权重。 所提出的RCMSTR在各种评估协议下展示了卓越的表现，超越了现有的最先进的自监督STR技术，在不同的STR相关下游任务中表现出色。消融研究和定性实验结果进一步验证了我们的方法的有效性。代码和预训练模型将于这个网址提供：[此处为链接]。
URL
https://arxiv.org/abs/2411.11219
PDF
-
A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
2024-11-17 18:52:06Raihan Kabir, Naznin Haque, Md Saiful Islam, Marium-E-JannatAbstract
Visual question answering (VQA) refers to the problem where, given an image and a natural language question about the image, a correct natural language answer has to be generated. A VQA model has to demonstrate both the visual understanding of the image and the semantic understanding of the question, demonstrating reasoning capability. Since the inception of this field, a plethora of VQA datasets and models have been published. In this article, we meticulously analyze the current state of VQA datasets and models, while cleanly dividing them into distinct categories and then summarizing the methodologies and characteristics of each category. We divide VQA datasets into four categories: (1) available datasets that contain a rich collection of authentic images, (2) synthetic datasets that contain only synthetic images produced through artificial means, (3) diagnostic datasets that are specially designed to test model performance in a particular area, e.g., understanding the scene text, and (4) KB (Knowledge-Based) datasets that are designed to measure a model's ability to utilize outside knowledge. Concurrently, we explore six main paradigms of VQA models: fusion, where we discuss different methods of fusing information between visual and textual modalities; attention, the technique of using information from one modality to filter information from another; external knowledge base, where we discuss different models utilizing outside information; composition or reasoning, where we analyze techniques to answer advanced questions that require complex reasoning steps; explanation, which is the process of generating visual and textual descriptions to verify sound reasoning; and graph models, which encode and manipulate relationships through nodes in a graph. We also discuss some miscellaneous topics, such as scene text understanding, counting, and bias reduction.
Abstract (translated)
视觉问答（VQA）指的是这样一个问题：给定一张图像和一个关于该图像的自然语言问题，需要生成正确的自然语言答案。一个VQA模型必须同时展示对图像的视觉理解和对问题的语义理解，并表现出推理能力。自这个领域诞生以来，大量的VQA数据集和模型已经被发表。本文详细分析了当前VQA数据集和模型的状态，清晰地将它们划分为不同的类别并总结每个类别的方法论和特点。我们将VQA数据集分为四类：（1）包含大量真实图像的现有数据集；（2）仅含有通过人工手段生成的合成图像的数据集；（3）专门设计用于测试特定领域模型性能的诊断数据集，例如理解场景文本；（4）知识基础(KB)数据集，旨在衡量模型利用外部知识的能力。同时，我们探索了VQA模型的六大主要范式：融合，讨论视觉和文本模式之间信息整合的不同方法；注意力，使用一种模态的信息来过滤另一种模态的信息的技术；外部知识库，探讨使用外部信息的不同模型；组合或推理，分析解答需要复杂推理步骤的高级问题的技术；解释，生成用于验证合理推理过程的视觉和文本描述的过程；以及图模型，通过图中的节点编码和操作关系。我们还讨论了一些杂项话题，如场景文本理解、计数和偏见减少。
URL
https://arxiv.org/abs/2411.11150
PDF
-
Partial Scene Text Retrieval
2024-11-15 15:08:04Hao Wang, Minghui Liao, Zhouyi Xie, Wenyu Liu, Xiang BaiAbstract
The task of partial scene text retrieval involves localizing and searching for text instances that are the same or similar to a given query text from an image gallery. However, existing methods can only handle text-line instances, leaving the problem of searching for partial patches within these text-line instances unsolved due to a lack of patch annotations in the training data. To address this issue, we propose a network that can simultaneously retrieve both text-line instances and their partial patches. Our method embeds the two types of data (query text and scene text instances) into a shared feature space and measures their cross-modal similarities. To handle partial patches, our proposed approach adopts a Multiple Instance Learning (MIL) approach to learn their similarities with query text, without requiring extra annotations. However, constructing bags, which is a standard step of conventional MIL approaches, can introduce numerous noisy samples for training, and lower inference speed. To address this issue, we propose a Ranking MIL (RankMIL) approach to adaptively filter those noisy samples. Additionally, we present a Dynamic Partial Match Algorithm (DPMA) that can directly search for the target partial patch from a text-line instance during the inference stage, without requiring bags. This greatly improves the search efficiency and the performance of retrieving partial patches. The source code and dataset are available at this https URL.
Abstract (translated)
部分场景文本检索的任务涉及从图像库中定位和搜索与给定查询文本相同或相似的文本实例。然而，现有的方法只能处理整行文本实例，由于训练数据中缺乏补丁标注，导致在这些文本行实例内搜索部分补丁的问题仍未解决。为了解决这一问题，我们提出了一种能够同时检索文本行实例及其部分补丁的网络。我们的方法将两类数据（查询文本和场景文本实例）嵌入到一个共享特征空间，并测量它们之间的跨模态相似性。为了处理部分补丁，我们采用多示例学习（MIL）的方法来学习其与查询文本的相似性，而无需额外标注。然而，构建包袋——这是传统MIL方法的标准步骤——可能会引入大量噪音样本进行训练，并降低推理速度。为了解决这一问题，我们提出了一种排名多实例学习（RankMIL）方法来自适应地过滤这些噪音样本。此外，我们还提出了动态部分匹配算法（DPMA），该算法可以在推理阶段直接从文本行实例中搜索目标部分补丁，而无需构建包袋。这大大提高了搜索效率和检索部分补丁的性能。源代码和数据集可在[此处](https://www.example.com)获得（请将此链接替换为实际提供的URL）。
URL
https://arxiv.org/abs/2411.10261
PDF
-
Real-Time Text Detection with Similar Mask in Traffic, Industrial, and Natural Scenes
2024-11-05 04:08:59Xu Han, Junyu Gao, Chuang Yang, Yuan Yuan, Qi WangAbstract
Texts on the intelligent transportation scene include mass information. Fully harnessing this information is one of the critical drivers for advancing intelligent transportation. Unlike the general scene, detecting text in transportation has extra demand, such as a fast inference speed, except for high accuracy. Most existing real-time text detection methods are based on the shrink mask, which loses some geometry semantic information and needs complex post-processing. In addition, the previous method usually focuses on correct output, which ignores feature correction and lacks guidance during the intermediate process. To this end, we propose an efficient multi-scene text detector that contains an effective text representation similar mask (SM) and a feature correction module (FCM). Unlike previous methods, the former aims to preserve the geometric information of the instances as much as possible. Its post-progressing saves 50$\%$ of the time, accurately and efficiently reconstructing text contours. The latter encourages false positive features to move away from the positive feature center, optimizing the predictions from the feature level. Some ablation studies demonstrate the efficiency of the SM and the effectiveness of the FCM. Moreover, the deficiency of existing traffic datasets (such as the low-quality annotation or closed source data unavailability) motivated us to collect and annotate a traffic text dataset, which introduces motion blur. In addition, to validate the scene robustness of the SM-Net, we conduct experiments on traffic, industrial, and natural scene datasets. Extensive experiments verify it achieves (SOTA) performance on several benchmarks. The code and dataset are available at: \url{this https URL}.
Abstract (translated)
智能交通场景中的文本包括大量信息，充分利用这些信息是推动智能交通发展的重要驱动力之一。与一般场景不同的是，在交通中检测文本还具有额外需求，除了高精度外还需要快速推理速度。现有的大多数实时文本检测方法都是基于缩放掩码的，这种方法会丢失一些几何语义信息，并且需要复杂的后处理。此外，之前的方法通常专注于正确输出，忽略了特征校正并在中间过程中缺乏指导。为此，我们提出了一种高效的多场景文本检测器，该检测器包含一种有效的文本表示相似掩码（SM）和一个特征校正模块（FCM）。与以前的方法不同的是，前者旨在尽可能保留实例的几何信息，其后处理节省了50%的时间，并且能够准确高效地重构文本轮廓。后者鼓励误报特征远离正样本特征中心，从特征级别优化预测结果。一些消融研究展示了SM的有效性和FCM的效果。此外，现有交通数据集（如低质量标注或封闭源数据不可用）的不足促使我们收集并标注了一个包含运动模糊的交通文本数据集。为了验证SM-Net在场景中的鲁棒性，我们在交通、工业和自然场景的数据集上进行了实验。广泛的实验证明它在多个基准测试中实现了最先进的（SOTA）性能。代码和数据集可在以下链接获取：\url{this https URL}。
URL
https://arxiv.org/abs/2411.02794
PDF
-
TextDestroyer: A Training- and Annotation-Free Diffusion Method for Destroying Anomal Text from Images
2024-11-01 04:41:00Mengcheng Li, Mingbao Lin, Fei Chao, Chia-Wen Lin, Rongrong JiAbstract
In this paper, we propose TextDestroyer, the first training- and annotation-free method for scene text destruction using a pre-trained diffusion model. Existing scene text removal models require complex annotation and retraining, and may leave faint yet recognizable text information, compromising privacy protection and content concealment. TextDestroyer addresses these issues by employing a three-stage hierarchical process to obtain accurate text masks. Our method scrambles text areas in the latent start code using a Gaussian distribution before reconstruction. During the diffusion denoising process, self-attention key and value are referenced from the original latent to restore the compromised background. Latent codes saved at each inversion step are used for replacement during reconstruction, ensuring perfect background restoration. The advantages of TextDestroyer include: (1) it eliminates labor-intensive data annotation and resource-intensive training; (2) it achieves more thorough text destruction, preventing recognizable traces; and (3) it demonstrates better generalization capabilities, performing well on both real-world scenes and generated images.
Abstract (translated)
在这篇论文中，我们提出了TextDestroyer，这是一种无需训练和标注的场景文本破坏方法，利用预训练扩散模型实现。现有的场景文本去除模型需要复杂的标注和重新训练，并且可能会留下模糊但可识别的文字信息，从而影响隐私保护和内容隐藏。TextDestroyer通过采用三阶段分层过程来获取准确的文本掩码，解决了这些问题。我们的方法在重建之前使用高斯分布对潜在代码中的文本区域进行扰乱。在扩散去噪过程中，引用原始潜变量的自注意力键和值以恢复受损背景。每个逆向步骤保存的潜在编码用于重建时替换，确保完美恢复背景。TextDestroyer的优势包括：(1) 消除了劳动密集型数据标注和资源密集型训练；(2) 实现了更彻底的文字破坏，防止可识别痕迹的存在；(3) 展示了更好的泛化能力，在真实场景图像和生成的图像上均表现良好。
URL
https://arxiv.org/abs/2411.00355
PDF
-
DiffSTR: Controlled Diffusion Models for Scene Text Removal
2024-10-29 04:20:21Sanhita Pathak, Vinay Kaushik, Brejesh LallAbstract
To prevent unauthorized use of text in images, Scene Text Removal (STR) has become a crucial task. It focuses on automatically removing text and replacing it with a natural, text-less background while preserving significant details such as texture, color, and contrast. Despite its importance in privacy protection, STR faces several challenges, including boundary artifacts, inconsistent texture and color, and preserving correct shadows. Most STR approaches estimate a text region mask to train a model, solving for image translation or inpainting to generate a text-free image. Thus, the quality of the generated image depends on the accuracy of the inpainting mask and the generator's capability. In this work, we leverage the superior capabilities of diffusion models in generating high-quality, consistent images to address the STR problem. We introduce a ControlNet diffusion model, treating STR as an inpainting task. To enhance the model's robustness, we develop a mask pretraining pipeline to condition our diffusion model. This involves training a masked autoencoder (MAE) using a combination of box masks and coarse stroke masks, and fine-tuning it using masks derived from our novel segmentation-based mask refinement framework. This framework iteratively refines an initial mask and segments it using the SLIC and Hierarchical Feature Selection (HFS) algorithms to produce an accurate final text mask. This improves mask prediction and utilizes rich textural information in natural scene images to provide accurate inpainting masks. Experiments on the SCUT-EnsText and SCUT-Syn datasets demonstrate that our method significantly outperforms existing state-of-the-art techniques.
Abstract (translated)
为了防止未经授权使用图像中的文本，场景文本移除（STR）已成为一项关键任务。它专注于自动删除文本并用自然无文字的背景替换，同时保留诸如纹理、颜色和对比度等重要细节。尽管在隐私保护方面具有重要意义，但STR面临多种挑战，包括边界伪影、不一致的纹理和色彩以及保持正确的阴影。大多数STR方法都会估计一个文本区域掩码来训练模型，通过解决图像翻译或补全问题生成无文字的图像。因此，生成图像的质量取决于补全过程掩码的准确性和生成器的能力。在这项工作中，我们利用扩散模型在生成高质量、一致图像方面的优越能力来解决STR问题。我们引入了一个ControlNet扩散模型，将STR视为一种补全任务。为了提高模型的鲁棒性，我们开发了一种掩码预训练管道来对我们的扩散模型进行条件设置。这包括使用框掩码和粗略笔触掩码组合训练一个掩码自动编码器（MAE），并用来自我们新颖分割基础的掩码优化框架产生的掩码对其进行微调。该框架通过迭代地精化初始掩码，并利用SLIC和层次特征选择（HFS）算法进行分段，生成准确的最终文本掩码。这提高了掩码预测精度，并利用自然场景图像中的丰富纹理信息提供精确的补全掩码。在SCUT-EnsText和SCUT-Syn数据集上的实验表明，我们的方法显著优于现有的最先进的技术。
URL
https://arxiv.org/abs/2410.21721
PDF
-
KhmerST: A Low-Resource Khmer Scene Text Detection and Recognition Benchmark
2024-10-23 21:04:24Vannkinh Nom, Souhail Bakkali, Muhammad Muzzamil Luqman, Micka\"el Coustaty, Jean-Marc OgierAbstract
Developing effective scene text detection and recognition models hinges on extensive training data, which can be both laborious and costly to obtain, especially for low-resourced languages. Conventional methods tailored for Latin characters often falter with non-Latin scripts due to challenges like character stacking, diacritics, and variable character widths without clear word boundaries. In this paper, we introduce the first Khmer scene-text dataset, featuring 1,544 expert-annotated images, including 997 indoor and 547 outdoor scenes. This diverse dataset includes flat text, raised text, poorly illuminated text, distant and partially obscured text. Annotations provide line-level text and polygonal bounding box coordinates for each scene. The benchmark includes baseline models for scene-text detection and recognition tasks, providing a robust starting point for future research endeavors. The KhmerST dataset is publicly accessible at this https URL.
Abstract (translated)
开发有效的场景文本检测和识别模型依赖于大量的训练数据，而获取这些数据既费时又昂贵，特别是对于资源匮乏的语言来说更是如此。针对拉丁字母的传统方法在处理非拉丁脚本时往往效果不佳，因为存在字符堆叠、附加符号以及没有明确单词边界的可变字符宽度等挑战。本文介绍了首个高棉语场景文本数据集，该数据集包含1,544张专家标注的图像，其中包括997张室内场景和547张室外场景。这个多样化的数据集涵盖了平面文本、凸起文本、光照条件不佳的文本、远处以及部分遮挡的文本。标注信息提供了每个场景的行级文本及多边形边界框坐标。基准测试包括了场景文本检测与识别任务的基础模型，为未来的科研工作提供了一个坚实的基础。KhmerST数据集可以在以下链接中公开访问：[此处是提供的URL]。
URL
https://arxiv.org/abs/2410.18277
PDF
-
Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models
2024-10-21 16:30:29Yufei Zhan, Hongyin Zhao, Yousong Zhu, Fan Yang, Ming Tang, Jinqiao WangAbstract
Large Multimodal Models (LMMs) have achieved significant breakthroughs in various vision-language and vision-centric tasks based on auto-regressive modeling. However, these models typically focus on either vision-centric tasks, such as visual grounding and region description, or vision-language tasks, like image caption and multi-scenario VQAs. None of the LMMs have yet comprehensively unified both types of tasks within a single model, as seen in Large Language Models in the natural language processing field. Furthermore, even with abundant multi-task instruction-following data, directly stacking these data for universal capabilities extension remains challenging. To address these issues, we introduce a novel multi-dimension curated and consolidated multimodal dataset, named CCMD-8M, which overcomes the data barriers of unifying vision-centric and vision-language tasks through multi-level data curation and multi-task consolidation. More importantly, we present Griffon-G, a general large multimodal model that addresses both vision-centric and vision-language tasks within a single end-to-end paradigm. Griffon-G resolves the training collapse issue encountered during the joint optimization of these tasks, achieving better training efficiency. Evaluations across multimodal benchmarks, general Visual Question Answering (VQA) tasks, scene text-centric VQA tasks, document-related VQA tasks, Referring Expression Comprehension, and object detection demonstrate that Griffon-G surpasses the advanced LMMs and achieves expert-level performance in complicated vision-centric tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2410.16163
PDF
-
Robust Loop Closure by Textual Cues in Challenging Environments
2024-10-21 10:54:38Tongxing Jin, Thien-Minh Nguyen, Xinhang Xu, Yizhuo Yang, Shenghai Yuan, Jianping Li, Lihua XieAbstract
Loop closure is an important task in robot navigation. However, existing methods mostly rely on some implicit or heuristic features of the environment, which can still fail to work in common environments such as corridors, tunnels, and warehouses. Indeed, navigating in such featureless, degenerative, and repetitive (FDR) environments would also pose a significant challenge even for humans, but explicit text cues in the surroundings often provide the best assistance. This inspires us to propose a multi-modal loop closure method based on explicit human-readable textual cues in FDR environments. Specifically, our approach first extracts scene text entities based on Optical Character Recognition (OCR), then creates a local map of text cues based on accurate LiDAR odometry and finally identifies loop closure events by a graph-theoretic scheme. Experiment results demonstrate that this approach has superior performance over existing methods that rely solely on visual and LiDAR sensors. To benefit the community, we release the source code and datasets at \url{this https URL}.
Abstract (translated)
URL
https://arxiv.org/abs/2410.15869
PDF
-
Neural Active Structure-from-Motion in Dark and Textureless Environment
2024-10-20 12:29:01Kazuto Ichimaru, Diego Thomas, Takafumi Iwaguchi, Hiroshi KawasakiAbstract
Active 3D measurement, especially structured light (SL) has been widely used in various fields for its robustness against textureless or equivalent surfaces by low light illumination. In addition, reconstruction of large scenes by moving the SL system has become popular, however, there have been few practical techniques to obtain the system's precise pose information only from images, since most conventional techniques are based on image features, which cannot be retrieved under textureless environments. In this paper, we propose a simultaneous shape reconstruction and pose estimation technique for SL systems from an image set where sparsely projected patterns onto the scene are observed (i.e. no scene texture information), which we call Active SfM. To achieve this, we propose a full optimization framework of the volumetric shape that employs neural signed distance fields (Neural-SDF) for SL with the goal of not only reconstructing the scene shape but also estimating the poses for each motion of the system. Experimental results show that the proposed method is able to achieve accurate shape reconstruction as well as pose estimation from images where only projected patterns are observed.
Abstract (translated)
URL
https://arxiv.org/abs/2410.15378
PDF
-
ViConsFormer: Constituting Meaningful Phrases of Scene Texts using Transformer-based Method in Vietnamese Text-based Visual Question Answering
2024-10-18 03:00:03Nghia Hieu Nguyen, Tho Thanh Quan, Ngan Luu-Thuy NguyenAbstract
Text-based VQA is a challenging task that requires machines to use scene texts in given images to yield the most appropriate answer for the given question. The main challenge of text-based VQA is exploiting the meaning and information from scene texts. Recent studies tackled this challenge by considering the spatial information of scene texts in images via embedding 2D coordinates of their bounding boxes. In this study, we follow the definition of meaning from linguistics to introduce a novel method that effectively exploits the information from scene texts written in Vietnamese. Experimental results show that our proposed method obtains state-of-the-art results on two large-scale Vietnamese Text-based VQA datasets. The implementation can be found at this link.
Abstract (translated)
URL
https://arxiv.org/abs/2410.14132
PDF
-
MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmark
2024-10-15 12:13:42Bin Shan, Xiang Fei, Wei Shi, An-Lan Wang, Guozhi Tang, Lei Liao, Jingqun Tang, Xiang Bai, Can HuangAbstract
The comprehension of text-rich visual scenes has become a focal point for evaluating Multi-modal Large Language Models (MLLMs) due to their widespread applications. Current benchmarks tailored to the scenario emphasize perceptual capabilities, while overlooking the assessment of cognitive abilities. To address this limitation, we introduce a Multimodal benchmark towards Text-rich visual scenes, to evaluate the Cognitive capabilities of MLLMs through visual reasoning and content-creation tasks (MCTBench). To mitigate potential evaluation bias from the varying distributions of datasets, MCTBench incorporates several perception tasks (e.g., scene text recognition) to ensure a consistent comparison of both the cognitive and perceptual capabilities of MLLMs. To improve the efficiency and fairness of content-creation evaluation, we conduct an automatic evaluation pipeline. Evaluations of various MLLMs on MCTBench reveal that, despite their impressive perceptual capabilities, their cognition abilities require enhancement. We hope MCTBench will offer the community an efficient resource to explore and enhance cognitive capabilities towards text-rich visual scenes.
Abstract (translated)
URL
https://arxiv.org/abs/2410.11538
PDF
-
Tex4D: Zero-shot 4D Scene Texturing with Video Diffusion Models
2024-10-14 17:59:59Jingzhi Bao, Xueting Li, Ming-Hsuan YangAbstract
3D meshes are widely used in computer vision and graphics for their efficiency in animation and minimal memory use, playing a crucial role in movies, games, AR, and VR. However, creating temporally consistent and realistic textures for mesh sequences remains labor-intensive for professional artists. On the other hand, while video diffusion models excel at text-driven video generation, they often lack 3D geometry awareness and struggle with achieving multi-view consistent texturing for 3D meshes. In this work, we present Tex4D, a zero-shot approach that integrates inherent 3D geometry knowledge from mesh sequences with the expressiveness of video diffusion models to produce multi-view and temporally consistent 4D textures. Given an untextured mesh sequence and a text prompt as inputs, our method enhances multi-view consistency by synchronizing the diffusion process across different views through latent aggregation in the UV space. To ensure temporal consistency, we leverage prior knowledge from a conditional video generation model for texture synthesis. However, straightforwardly combining the video diffusion model and the UV texture aggregation leads to blurry results. We analyze the underlying causes and propose a simple yet effective modification to the DDIM sampling process to address this issue. Additionally, we introduce a reference latent texture to strengthen the correlation between frames during the denoising process. To the best of our knowledge, Tex4D is the first method specifically designed for 4D scene texturing. Extensive experiments demonstrate its superiority in producing multi-view and multi-frame consistent videos based on untextured mesh sequences.
Abstract (translated)
URL
https://arxiv.org/abs/2410.10821
PDF
-
First Creating Backgrounds Then Rendering Texts: A New Paradigm for Visual Text Blending
2024-10-14 05:23:43Zhenhang Li, Yan Shu, Weichao Zeng, Dongbao Yang, Yu ZhouAbstract
Diffusion models, known for their impressive image generation abilities, have played a pivotal role in the rise of visual text generation. Nevertheless, existing visual text generation methods often focus on generating entire images with text prompts, leading to imprecise control and limited practicality. A more promising direction is visual text blending, which focuses on seamlessly merging texts onto text-free backgrounds. However, existing visual text blending methods often struggle to generate high-fidelity and diverse images due to a shortage of backgrounds for synthesis and limited generalization capabilities. To overcome these challenges, we propose a new visual text blending paradigm including both creating backgrounds and rendering texts. Specifically, a background generator is developed to produce high-fidelity and text-free natural images. Moreover, a text renderer named GlyphOnly is designed for achieving visually plausible text-background integration. GlyphOnly, built on a Stable Diffusion framework, utilizes glyphs and backgrounds as conditions for accurate rendering and consistency control, as well as equipped with an adaptive text block exploration strategy for small-scale text rendering. We also explore several downstream applications based on our method, including scene text dataset synthesis for boosting scene text detectors, as well as text image customization and editing. Code and model will be available at \url{this https URL}.
Abstract (translated)
URL
https://arxiv.org/abs/2410.10168
PDF
-
Signage-Aware Exploration in Open World using Venue Maps
2024-10-14 04:18:25Chang Chen, Liang Lu, Lei Yang, Yinqiang Zhang, Yizhou Chen, Ruixing Jia, Jia PanAbstract
Current exploration methods struggle to search for shops in unknown open-world environments due to a lack of prior knowledge and text recognition capabilities. Venue maps offer valuable information that can aid exploration planning by correlating scene signage with map data. However, the arbitrary shapes and styles of the text on signage, along with multi-view inconsistencies, pose significant challenges for accurate recognition by robots. Additionally, the discrepancies between real-world environments and venue maps hinder the incorporation of text information into planners. This paper introduces a novel signage-aware exploration system to address these challenges, enabling the robot to utilize venue maps effectively. We propose a signage understanding method that accurately detects and recognizes the text on signage using a diffusion-based text instance retrieval method combined with a 2D-to-3D semantic fusion strategy. Furthermore, we design a venue map-guided exploration-exploitation planner that balances exploration in unknown regions using a directional heuristic derived from venue maps with exploitation to get close and adjust orientation for better recognition. Experiments in large-scale shopping malls demonstrate our method's superior signage recognition accuracy and coverage efficiency, outperforming state-of-the-art scene text spotting methods and traditional exploration methods.
Abstract (translated)
URL
https://arxiv.org/abs/2410.10143
PDF
-
TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control
2024-10-14 03:50:39Weichao Zeng, Yan Shu, Zhenhang Li, Dongbao Yang, Yu ZhouAbstract
Centred on content modification and style preservation, Scene Text Editing (STE) remains a challenging task despite considerable progress in text-to-image synthesis and text-driven image manipulation recently. GAN-based STE methods generally encounter a common issue of model generalization, while Diffusion-based STE methods suffer from undesired style deviations. To address these problems, we propose TextCtrl, a diffusion-based method that edits text with prior guidance control. Our method consists of two key components: (i) By constructing fine-grained text style disentanglement and robust text glyph structure representation, TextCtrl explicitly incorporates Style-Structure guidance into model design and network training, significantly improving text style consistency and rendering accuracy. (ii) To further leverage the style prior, a Glyph-adaptive Mutual Self-attention mechanism is proposed which deconstructs the implicit fine-grained features of the source image to enhance style consistency and vision quality during inference. Furthermore, to fill the vacancy of the real-world STE evaluation benchmark, we create the first real-world image-pair dataset termed ScenePair for fair comparisons. Experiments demonstrate the effectiveness of TextCtrl compared with previous methods concerning both style fidelity and text accuracy.
Abstract (translated)
URL
https://arxiv.org/abs/2410.10133
PDF
