Abstract
Visual question answering (VQA) refers to the problem where, given an image and a natural language question about the image, a correct natural language answer has to be generated. A VQA model has to demonstrate both the visual understanding of the image and the semantic understanding of the question, demonstrating reasoning capability. Since the inception of this field, a plethora of VQA datasets and models have been published. In this article, we meticulously analyze the current state of VQA datasets and models, while cleanly dividing them into distinct categories and then summarizing the methodologies and characteristics of each category. We divide VQA datasets into four categories: (1) available datasets that contain a rich collection of authentic images, (2) synthetic datasets that contain only synthetic images produced through artificial means, (3) diagnostic datasets that are specially designed to test model performance in a particular area, e.g., understanding the scene text, and (4) KB (Knowledge-Based) datasets that are designed to measure a model's ability to utilize outside knowledge. Concurrently, we explore six main paradigms of VQA models: fusion, where we discuss different methods of fusing information between visual and textual modalities; attention, the technique of using information from one modality to filter information from another; external knowledge base, where we discuss different models utilizing outside information; composition or reasoning, where we analyze techniques to answer advanced questions that require complex reasoning steps; explanation, which is the process of generating visual and textual descriptions to verify sound reasoning; and graph models, which encode and manipulate relationships through nodes in a graph. We also discuss some miscellaneous topics, such as scene text understanding, counting, and bias reduction.
Abstract (translated)
视觉问答（VQA）指的是这样一个问题：给定一张图像和一个关于该图像的自然语言问题，需要生成正确的自然语言答案。一个VQA模型必须同时展示对图像的视觉理解和对问题的语义理解，并表现出推理能力。自这个领域诞生以来，大量的VQA数据集和模型已经被发表。本文详细分析了当前VQA数据集和模型的状态，清晰地将它们划分为不同的类别并总结每个类别的方法论和特点。我们将VQA数据集分为四类：（1）包含大量真实图像的现有数据集；（2）仅含有通过人工手段生成的合成图像的数据集；（3）专门设计用于测试特定领域模型性能的诊断数据集，例如理解场景文本；（4）知识基础(KB)数据集，旨在衡量模型利用外部知识的能力。同时，我们探索了VQA模型的六大主要范式：融合，讨论视觉和文本模式之间信息整合的不同方法；注意力，使用一种模态的信息来过滤另一种模态的信息的技术；外部知识库，探讨使用外部信息的不同模型；组合或推理，分析解答需要复杂推理步骤的高级问题的技术；解释，生成用于验证合理推理过程的视觉和文本描述的过程；以及图模型，通过图中的节点编码和操作关系。我们还讨论了一些杂项话题，如场景文本理解、计数和偏见减少。
URL
https://arxiv.org/abs/2411.11150
