Abstract
This report evaluates PDF-to-Markdown conversion using recent Vision-Language Models (VLMs) on challenging French documents. Document parsing is a critical step for Retrieval-Augmented Generation (RAG) pipelines, where transcription and layout errors propagate to downstream retrieval and grounding. Existing benchmarks often emphasize English or Chinese and can over-penalize benign formatting and linearization choices (e.g., line breaks, list segmentation, alternative table renderings) that are largely irrelevant for downstream use. We introduce a French-focused benchmark of difficult pages selected via model-disagreement sampling from a corpus of 60{,}000 documents, covering handwritten forms, complex layouts, dense tables, and graphics-rich pages. Evaluation is performed with unit-test-style checks that target concrete failure modes (text presence, reading order, and local table constraints) combined with category-specific normalization designed to discount presentation-only variance. Across 15 models, we observe substantially higher robustness for the strongest proprietary models on handwriting and forms, while several open-weights systems remain competitive on standard printed layouts.
Abstract (translated)
该报告评估了使用近期的视觉语言模型（VLMs）对具有挑战性的法语文档进行PDF到Markdown转换的效果。文档解析是检索增强生成（RAG）管道中的关键步骤，其中转录和布局错误会传播到下游的检索和定位任务中。现有的基准测试通常更侧重于英语或中文，并且往往会过度惩罚那些对于下游使用实际上无关紧要但与格式化选择相关的良性情况（例如，换行符、列表分割以及表格的不同呈现方式）。为此，我们引入了一个以法语为中心的新基准，该基准包含从60,000份文档中通过模型意见分歧采样得到的困难页面样本。这些样本涵盖了手写表单、复杂布局、密集表格和图形丰富的页面等不同情况。 评估是使用类似单元测试的方法进行的，包括针对具体失败模式（如文本存在性、阅读顺序以及局部表格约束）的检查，并结合了根据类别定制的标准化处理方法来消除仅与呈现有关的变化。在15种模型中，我们观察到最强劲的专有模型在手写和表单解析方面表现出显著更高的稳健性，而几种开源权重系统则在标准印刷布局上依然保持竞争力。
URL
https://arxiv.org/abs/2602.11960
