Abstract
Japanese scene text poses challenges that multilingual benchmarks often fail to capture, including mixed scripts, frequent vertical writing, and a character inventory far larger than the Latin alphabet. Although Japanese is included in several multilingual benchmarks, these resources do not adequately capture the language-specific complexities. Meanwhile, existing Japanese visual text datasets have primarily focused on scanned documents, leaving in-the-wild scene text underexplored. To fill this gap, we introduce JaWildText, a diagnostic benchmark for evaluating vision-language models (VLMs) on Japanese scene text understanding. JaWildText contains 3,241 instances from 2,961 images newly captured in Japan, with 1.12 million annotated characters spanning 3,643 unique character types. It comprises three complementary tasks that vary in visual organization, output format, and writing style: (i) Dense Scene Text Visual Question Answering (STVQA), which requires reasoning over multiple pieces of visual text evidence; (ii) Receipt Key Information Extraction (KIE), which tests layout-aware structured extraction from mobile-captured receipts; and (iii) Handwriting OCR, which evaluates page-level transcription across various media and writing directions. We evaluate 14 open-weight VLMs and find that the best model achieves an average score of 0.64 across the three tasks. Error analyses show recognition remains the dominant bottleneck, especially for kanji. JaWildText enables fine-grained, script-aware diagnosis of Japanese scene text capabilities, and will be released with evaluation code.
Abstract (translated)
日语场景文本面临着多语言基准测试常难以捕捉的挑战，包括混合文字系统、频繁的竖排书写，以及远超拉丁字母的字符总量。尽管日语已被纳入多个多语言基准测试，但这些资源未能充分体现该语言特有的复杂性。与此同时，现有日语视觉文本数据集主要聚焦扫描文档，对现实场景中的文本探索不足。为填补这一空白，我们推出JaWildText——一个用于评估视觉-语言模型（VLMs）日语场景文本理解能力的诊断基准。该数据集包含2,961张在日本新采集图像中的3,241个实例，覆盖3,643种独特字符类型，共标注112万个字符。数据集包含三个互补任务，在视觉组织、输出格式和书写风格上各有差异：（i）密集场景文本视觉问答（STVQA），需对多段视觉文本证据进行推理；（ii）收据关键信息提取（KIE），测试对手机拍摄收据的布局感知结构化提取；（iii）手写OCR，评估跨媒介和书写方向的整页转录。我们评估了14个开源视觉-语言模型，发现最佳模型在三项任务上的平均得分仅为0.64。错误分析表明，识别仍是主要瓶颈，尤其是对汉字。JaWildText支持细粒度、感知文字系统的日语场景文本能力诊断，并将随评估代码一并开源。
URL
https://arxiv.org/abs/2603.27942
