Abstract
Designing datasets for Visual Question Answering (VQA) is a difficult and complex task that requires NLP for parsing and computer vision for analysing the relevant aspects of the image for answering the question asked. Several benchmark datasets have been developed by researchers but there are many issues with using them for methodical performance tests. This paper proposes a new benchmark dataset -- a pilot version called VQA-Levels is ready now -- for testing VQA systems systematically and assisting researchers in advancing the field. The questions are classified into seven levels ranging from direct answers based on low-level image features (without needing even a classifier) to those requiring high-level abstraction of the entire image content. The questions in the dataset exhibit one or many of ten properties. Each is categorised into a specific level from 1 to 7. Levels 1 - 3 are directly on the visual content while the remaining levels require extra knowledge about the objects in the image. Each question generally has a unique one or two-word answer. The questions are 'natural' in the sense that a human is likely to ask such a question when seeing the images. An example question at Level 1 is, ``What is the shape of the red colored region in the image?" while at Level 7, it is, ``Why is the man cutting the paper?". Initial testing of the proposed dataset on some of the existing VQA systems reveals that their success is high on Level 1 (low level features) and Level 2 (object classification) questions, least on Level 3 (scene text) followed by Level 6 (extrapolation) and Level 7 (whole scene analysis) questions. The work in this paper will go a long way to systematically analyze VQA systems.
Abstract (translated)
设计用于视觉问答（VQA）的数据集是一项困难且复杂的任务，需要自然语言处理技术来解析问题，并使用计算机视觉技术分析图像中与回答问题相关的方面。尽管研究人员已经开发出了几个基准数据集，但在进行系统性性能测试时这些数据集仍然存在诸多问题。本文提出了一种新的基准数据集——一个称为VQA-Levels的试点版本现已准备就绪——旨在对VQA系统进行系统的测试，并帮助研究者推进该领域的进展。这些问题被分类为七个等级，从基于图像低级特征（甚至不需要使用分类器）直接回答的问题到需要整个图像内容高层次抽象的问题不等。数据集中的问题表现出十种属性之一或多种。每一个问题根据其复杂性从1到7进行分级。等级1至3主要涉及视觉内容的直接分析，而剩余的等级则需要关于图像中物体的额外知识。一般而言，每个问题通常具有一个或两个词的独特答案。这些问题在“自然”方面表现出色，也就是说当看到这些图像时人类有可能会提出这样的问题。例如，在Level 1（低级特征）的问题是：“图片中的红色区域是什么形状？”而在Level 7（整个场景分析），问题是：“为什么这个男人正在剪纸？”对现有VQA系统进行的初步测试显示，它们在Level 1和Level 2（对象分类）这类简单问题上的成功率较高，在Level 3（场景文本）、Level 6（推断）以及最难的Level 7（整个场景分析）类问题上的表现则相对较差。本文的研究将有助于对VQA系统进行系统的分析。
URL
https://arxiv.org/abs/2502.02951
