Abstract
Multimodal Large Language Models (MLLMs) have demonstrated significant capabilities in joint visual and linguistic tasks. However, existing Visual Question Answering (VQA) benchmarks often fail to evaluate deep semantic understanding, particularly in complex domains like visual art analysis. Confined to simple syntactic structures and surface-level attributes, these questions fail to capture the diversity and depth of human visual inquiry. This limitation incentivizes models to exploit statistical shortcuts rather than engage in visual reasoning. To address this gap, we introduce VQArt-Bench, a new, large-scale VQA benchmark for the cultural heritage domain. This benchmark is constructed using a novel multi-agent pipeline where specialized agents collaborate to generate nuanced, validated, and linguistically diverse questions. The resulting benchmark is structured along relevant visual understanding dimensions that probe a model's ability to interpret symbolic meaning, narratives, and complex visual relationships. Our evaluation of 14 state-of-the-art MLLMs on this benchmark reveals significant limitations in current models, including a surprising weakness in simple counting tasks and a clear performance gap between proprietary and open-source models.
Abstract (translated)
多模态大型语言模型（MLLMs）在视觉和语言任务的联合处理方面展示了显著的能力。然而，现有的视觉问答（VQA）基准测试常常无法评估深层语义理解能力，尤其是在像视觉艺术分析这样复杂的领域中。这些问题局限于简单的句法结构和表面属性，未能捕捉到人类视觉查询的多样性和深度。这种局限性促使模型利用统计捷径而不是进行视觉推理。为了解决这一差距，我们引入了VQArt-Bench，这是一个新的大规模VQA基准测试工具，专门针对文化遗产领域设计。该基准测试是通过一个新颖的多代理管道构建的，在这个管道中，专业代理人合作生成细腻、验证过的和语言多样化的提问。由此产生的基准测试涵盖了相关的视觉理解维度，以考察模型解读象征意义、叙述以及复杂视觉关系的能力。我们在这一基准上对14个最先进的MLLMs进行了评估，结果揭示了现有模型中存在的显著局限性，包括在简单的计数任务中的意外薄弱表现，以及专有和开源模型之间的明显性能差距。
URL
https://arxiv.org/abs/2510.12750
