Abstract
Mainstream Scene Text Recognition (STR) algorithms are developed based on RGB cameras which are sensitive to challenging factors such as low illumination, motion blur, and cluttered backgrounds. In this paper, we propose to recognize the scene text using bio-inspired event cameras by collecting and annotating a large-scale benchmark dataset, termed EventSTR. It contains 9,928 high-definition (1280 * 720) event samples and involves both Chinese and English characters. We also benchmark multiple STR algorithms as the baselines for future works to compare. In addition, we propose a new event-based scene text recognition framework, termed SimC-ESTR. It first extracts the event features using a visual encoder and projects them into tokens using a Q-former module. More importantly, we propose to augment the vision tokens based on a memory mechanism before feeding into the large language models. A similarity-based error correction mechanism is embedded within the large language model to correct potential minor errors fundamentally based on contextual information. Extensive experiments on the newly proposed EventSTR dataset and two simulation STR datasets fully demonstrate the effectiveness of our proposed model. We believe that the dataset and algorithmic model can innovatively propose an event-based STR task and are expected to accelerate the application of event cameras in various industries. The source code and pre-trained models will be released on this https URL
Abstract (translated)
主流的场景文本识别（STR）算法基于RGB相机开发，这些相机对诸如低光照、运动模糊和杂乱背景等挑战性因素非常敏感。在本文中，我们提出了一种使用仿生事件相机来识别场景文本的方法，并为此收集并标注了一个大规模基准数据集，称为EventSTR。该数据集中包含9,928个高清（1280 * 720）事件样本，涉及中文和英文字符。此外，我们还为未来的工作提供了多个STR算法作为基线模型以供比较。另外，我们提出了一种新的基于事件的场景文本识别框架，称为SimC-ESTR。该框架首先使用视觉编码器提取事件特征，并利用Q-former模块将这些特征映射到令牌中。更重要的是，在输入大型语言模型之前，我们提出了通过内存机制来增强视觉令牌的方法。此外，我们在大型语言模型中嵌入了一种基于相似性的错误纠正机制，可以根据上下文信息从根本上纠正潜在的小错误。在新提出的EventSTR数据集和两个模拟的STR数据集上的大量实验充分证明了我们所提出模型的有效性。我们认为，这个数据集和算法模型可以创新地提出一种基于事件的STR任务，并有望加速事件相机在各个行业的应用。源代码和预训练模型将在以下网址发布：[https URL]
URL
https://arxiv.org/abs/2502.09020
