Abstract
Enterprise documents, such as forms and reports, embed critical information for downstream applications like data archiving, automated workflows, and analytics. Although generalist Vision Language Models (VLMs) perform well on established document understanding benchmarks, their ability to conduct holistic, fine-grained structured extraction across diverse document types and flexible schemas is not well studied. Existing Key Entity Extraction (KEE), Relation Extraction (RE), and Visual Question Answering (VQA) datasets are limited by narrow entity ontologies, simple queries, or homogeneous document types, often overlooking the need for adaptable and structured extraction. To address these gaps, we introduce ExStrucTiny, a new benchmark dataset for structured Information Extraction (IE) from document images, unifying aspects of KEE, RE, and VQA. Built through a novel pipeline combining manual and synthetic human-validated samples, ExStrucTiny covers more varied document types and extraction scenarios. We analyze open and closed VLMs on this benchmark, highlighting challenges such as schema adaptation, query under-specification, and answer localization. We hope our work provides a bedrock for improving generalist models for structured IE in documents.
Abstract (translated)
企业文档（如表格和报告）包含下游应用（例如数据归档、自动化工作流和数据分析）所需的至关重要的信息。尽管通用的视觉语言模型（VLMs）在已建立的文档理解基准测试中表现良好，但它们在整个多样化文档类型和灵活模式中的整体细粒度结构化提取能力尚未得到充分研究。现有的关键实体抽取（KEE）、关系抽取（RE）和视觉问答（VQA）数据集受限于狭窄的实体本体、简单的问题或单一类型的文件，往往忽略了对适应性强且具有结构性提取的需求。为了解决这些问题，我们引入了ExStrucTiny，这是一个新的基准数据集，用于从文档图像中进行结构化信息抽取，将KEE、RE和VQA的方面统一起来。通过结合手动和合成的人类验证样本的新颖管道构建而成，ExStrucTiny涵盖了更多种类的文件类型和提取场景。我们在该基准测试上分析了开放和封闭的VLMs，并指出了模式适应性差、问题描述不明确以及答案定位等挑战。我们希望我们的工作为提高文档中的结构化信息抽取的通用模型提供基础。
URL
https://arxiv.org/abs/2602.12203
