Abstract
Understanding signboard text in natural scenes is essential for real-world applications of Visual Question Answering (VQA), yet remains underexplored, particularly in low-resource languages. We introduce ViSignVQA, the first large-scale Vietnamese dataset designed for signboard-oriented VQA, which comprises 10,762 images and 25,573 question-answer pairs. The dataset captures the diverse linguistic, cultural, and visual characteristics of Vietnamese signboards, including bilingual text, informal phrasing, and visual elements such as color and layout. To benchmark this task, we adapted state-of-the-art VQA models (e.g., BLIP-2, LaTr, PreSTU, and SaL) by integrating a Vietnamese OCR model (SwinTextSpotter) and a Vietnamese pretrained language model (ViT5). The experimental results highlight the significant role of the OCR-enhanced context, with F1-score improvements of up to 209% when the OCR text is appended to questions. Additionally, we propose a multi-agent VQA framework combining perception and reasoning agents with GPT-4, achieving 75.98% accuracy via majority voting. Our study presents the first large-scale multimodal dataset for Vietnamese signboard understanding. This underscores the importance of domain-specific resources in enhancing text-based VQA for low-resource languages. ViSignVQA serves as a benchmark capturing real-world scene text characteristics and supporting the development and evaluation of OCR-integrated VQA models in Vietnamese.
Abstract (translated)
理解自然场景中的招牌文字对于视觉问答（Visual Question Answering，VQA）在现实世界的应用至关重要，然而这一领域尤其在低资源语言中仍然研究不足。我们介绍了ViSignVQA，这是首个为面向招牌的VQA设计的大规模越南语数据集，包含10,762张图像和25,573组问题-答案对。该数据集捕捉了越南语招牌文字的语言、文化和视觉特点，包括双语文本、非正式表达方式以及颜色和布局等视觉元素。 为了评估这一任务，我们通过整合一种越南语OCR模型（SwinTextSpotter）和一种预训练的越南语语言模型（ViT5），对最先进的VQA模型进行了改造，如BLIP-2、LaTr、PreSTU 和 SaL。实验结果表明，增强型 OCR 上下文发挥了显著作用，在将OCR 文本附加到问题时，F1 分数最高提高了 209%。 此外，我们提出了一种结合感知和推理代理以及GPT-4的多智能体VQA框架，通过多数投票达到了75.98%的准确率。我们的研究首次展示了大规模的越南语招牌理解多模态数据集，并突显了领域特定资源在增强基于文本的低资源语言 VQA 中的重要性。 ViSignVQA 作为基准，捕捉了现实世界场景文字的特点，并支持集成 OCR 的 VQA 模型在越南语中的开发和评估。
URL
https://arxiv.org/abs/2512.22218
