Abstract
Scene text recognition (STR) and handwritten text recognition (HTR) face significant challenges in accurately transcribing textual content from images into machine-readable formats. Conventional OCR models often predict transcriptions directly, which limits detailed reasoning about text structure. We propose a VQA-inspired data augmentation framework that strengthens OCR training through structured question-answering tasks. For each image-text pair, we generate natural-language questions probing character-level attributes such as presence, position, and frequency, with answers derived from ground-truth text. These auxiliary tasks encourage finer-grained reasoning, and the OCR model aligns visual features with textual queries to jointly reason over images and questions. Experiments on WordArt and Esposalles datasets show consistent improvements over baseline models, with significant reductions in both CER and WER. Our code is publicly available at this https URL.
Abstract (translated)
场景文本识别（STR）和手写文本识别（HTR）在将图像中的文字内容准确转录为机器可读格式时面临着重大挑战。传统的OCR模型通常直接预测转录结果，这限制了对文本结构的详细推理能力。我们提出了一种受VQA（视觉问答）启发的数据增强框架，该框架通过结构化的问答任务加强OCR训练。对于每个图像-文本配对，我们生成自然语言问题来探究字符级别的属性，如存在性、位置和频率，并根据真实文本信息得出答案。这些辅助任务鼓励更精细的推理过程，同时OCR模型将视觉特征与文本查询相结合，共同处理图像和问题以进行综合推理。在WordArt和Esposalles数据集上的实验表明，相较于基准模型，我们的方法在字符错误率（CER）和单词错误率（WER）上都有显著降低。我们公开可用的代码托管在此链接：[此URL]。
URL
https://arxiv.org/abs/2603.03580
