Abstract
Traditional machine learning models for Handwritten Text Recognition (HTR) rely on supervised training, requiring extensive manual annotations, and often produce errors due to the separation between layout and text processing. In contrast, Multimodal Large Language Models (MLLMs) offer a general approach to recognizing diverse handwriting styles without the need for model-specific training. The study benchmarks various proprietary and open-source LLMs against Transkribus models, evaluating their performance on both modern and historical datasets written in English, French, German, and Italian. In addition, emphasis is placed on testing the models' ability to autonomously correct previously generated outputs. Findings indicate that proprietary models, especially Claude 3.5 Sonnet, outperform open-source alternatives in zero-shot settings. MLLMs achieve excellent results in recognizing modern handwriting and exhibit a preference for the English language due to their pre-training dataset composition. Comparisons with Transkribus show no consistent advantage for either approach. Moreover, LLMs demonstrate limited ability to autonomously correct errors in zero-shot transcriptions.
Abstract (translated)
传统的手写文本识别（HTR）模型依赖于监督训练，需要大量的手动注释，并且由于布局处理和文字处理的分离常常会产生错误。相比之下，多模态大型语言模型（MLLMs）提供了一种通用的方法来识别多样化的书写风格，而无需特定于模型的训练。这项研究对各种专有和开源的大规模语言模型与Transkribus模型进行了基准测试，并评估了它们在现代和历史数据集上的表现，这些数据集包含英文、法文、德文和意大利文的手写文本。此外，该研究还强调了测试模型自主纠正先前生成的输出的能力。 研究结果表明，在零样本设置中，专有模型（特别是Claude 3.5 Sonnet）优于开源替代品。MLLMs在识别现代手写方面表现出色，并且由于其预训练数据集的构成而对英语表现出偏好。与Transkribus相比，没有一种方法显示出持续的优势。此外，在零样本转录中，LLMs自主纠正错误的能力有限。 简而言之，这项研究展示了MLLMs在处理多样化书写风格和现代手写方面的优越性能，并揭示了这些模型在不同语言背景下存在的局限性以及它们纠正自身错误能力的不足之处。
URL
https://arxiv.org/abs/2503.15195
