Abstract
In the field of scene text spotting, previous OCR methods primarily relied on image encoders and pre-trained text information, but they often overlooked the advantages of incorporating human language instructions. To address this gap, we propose InstructOCR, an innovative instruction-based scene text spotting model that leverages human language instructions to enhance the understanding of text within images. Our framework employs both text and image encoders during training and inference, along with instructions meticulously designed based on text attributes. This approach enables the model to interpret text more accurately and flexibly. Extensive experiments demonstrate the effectiveness of our model and we achieve state-of-the-art results on widely used benchmarks. Furthermore, the proposed framework can be seamlessly applied to scene text VQA tasks. By leveraging instruction strategies during pre-training, the performance on downstream VQA tasks can be significantly improved, with a 2.6% increase on the TextVQA dataset and a 2.1% increase on the ST-VQA dataset. These experimental results provide insights into the benefits of incorporating human language instructions for OCR-related tasks.
Abstract (translated)
在场景文本检测领域，之前的OCR方法主要依赖图像编码器和预训练的文本信息，但它们往往忽视了结合人类语言指令的优势。为了解决这一问题，我们提出了InstructOCR，这是一个创新性的基于指令的场景文本检测模型，利用人类语言指令来增强对图像中文本的理解。我们的框架在训练和推理过程中使用了文本编码器和图像编码器，并且根据文本属性精心设计了指令。这种方法使模型能够更准确、灵活地解释文本内容。广泛的实验表明我们模型的有效性，并且我们在广泛使用的基准测试中取得了最先进的结果。此外，所提出的框架可以无缝应用于场景文本视觉问答（VQA）任务。通过在预训练过程中利用指令策略，下游VQA任务的性能得到了显著提升，在TextVQA数据集上提升了2.6%，在ST-VQA数据集上提升了2.1%。这些实验结果为将人类语言指令融入OCR相关任务中的益处提供了见解。
URL
https://arxiv.org/abs/2412.15523
