Abstract
Handwritten text recognition (HTR) remains a challenging task, particularly for multi-page documents where pages share common formatting and contextual features. While modern optical character recognition (OCR) engines are proficient with printed text, their performance on handwriting is limited, often requiring costly labeled data for fine-tuning. In this paper, we explore the use of multi-modal large language models (MLLMs) for transcribing multi-page handwritten documents in a zero-shot setting. We investigate various configurations of commercial OCR engines and MLLMs, utilizing the latter both as end-to-end transcribers and as post-processors, with and without image components. We propose a novel method, '+first page', which enhances MLLM transcription by providing the OCR output of the entire document along with just the first page image. This approach leverages shared document features without incurring the high cost of processing all images. Experiments on a multi-page version of the IAM Handwriting Database demonstrate that '+first page' improves transcription accuracy, balances cost with performance, and even enhances results on out-of-sample text by extrapolating formatting and OCR error patterns from a single page.
Abstract (translated)
手写文本识别（HTR）仍然是一个具有挑战性的任务，特别是在多页文档中，这些文档共享相同的格式和上下文特征。尽管现代光学字符识别（OCR）引擎在处理印刷文本方面表现出色，但它们对手写文本的性能有限，并且通常需要昂贵的标注数据来进行微调。在这篇论文中，我们探讨了使用多模态大型语言模型（MLLMs）以零样本设置转录多页手写文档的方法。我们研究了各种商业OCR引擎和MLLM配置，利用后者作为端到端转录器以及有或没有图像组件的后处理工具。 我们提出了一种新颖的方法，“+首页面”，通过提供整个文档的OCR输出加上仅第一张图片来增强MLLM转录效果。这种方法利用了共享的文档特征，同时避免了处理所有图像所需的高昂成本。在IAM手写数据库的多页版本上的实验表明，“+首页面”方法提高了转录准确性，并且能够在不增加成本的情况下平衡性能；此外，在样本外文本上，该方法还能通过从单张图片中推断格式和OCR错误模式来提高结果质量。
URL
https://arxiv.org/abs/2502.20295
