Abstract
Multimodal large language models (MLLMs) have shown impressive capabilities across various domains, excelling in processing and understanding information from multiple modalities. Despite the rapid progress made previously, insufficient OCR ability hinders MLLMs from excelling in text-related tasks. In this paper, we present \textbf{Ocean-OCR}, a 3B MLLM with state-of-the-art performance on various OCR scenarios and comparable understanding ability on general tasks. We employ Native Resolution ViT to enable variable resolution input and utilize a substantial collection of high-quality OCR datasets to enhance the model performance. We demonstrate the superiority of Ocean-OCR through comprehensive experiments on open-source OCR benchmarks and across various OCR scenarios. These scenarios encompass document understanding, scene text recognition, and handwritten recognition, highlighting the robust OCR capabilities of Ocean-OCR. Note that Ocean-OCR is the first MLLM to outperform professional OCR models such as TextIn and PaddleOCR.
Abstract (translated)
多模态大型语言模型（MLLM）在多个领域展示了令人印象深刻的性能，特别是在处理和理解来自多种模式的信息方面表现出色。尽管之前取得了快速进展，但不足的光学字符识别（OCR）能力阻碍了MLLM在与文本相关任务中的表现。在这篇论文中，我们介绍了**Ocean-OCR**，这是一种3B参数大小的多模态语言模型，在各种OCR场景下展现出最先进的性能，并且在一般任务上具有相当的理解能力。为了使模型能够接受不同分辨率的输入，我们采用了原生分辨率ViT，并利用大量的高质量OCR数据集来提升模型的表现力。通过在开源OCR基准测试和多种OCR场景上的全面实验，展示了Ocean-OCR的优势。这些场景包括文档理解、场景文本识别以及手写识别等，突显了Ocean-OCR强大的OCR能力。值得注意的是，Ocean-OCR是首个在其性能上超过专业OCR模型（如TextIn和PaddleOCR）的多模态大型语言模型。
URL
https://arxiv.org/abs/2501.15558
