Abstract
Video text-based visual question answering (Video TextVQA) is a practical task that aims to answer questions by jointly reasoning textual and visual information in a given video. Inspired by the development of TextVQA in image domain, existing Video TextVQA approaches leverage a language model (e.g. T5) to process text-rich multiple frames and generate answers auto-regressively. Nevertheless, the spatio-temporal relationships among visual entities (including scene text and objects) will be disrupted and models are susceptible to interference from unrelated information, resulting in irrational reasoning and inaccurate answering. To tackle these challenges, we propose the TEA (stands for ``\textbf{T}rack th\textbf{E} \textbf{A}nswer'') method that better extends the generative TextVQA framework from image to video. TEA recovers the spatio-temporal relationships in a complementary way and incorporates OCR-aware clues to enhance the quality of reasoning questions. Extensive experiments on several public Video TextVQA datasets validate the effectiveness and generalization of our framework. TEA outperforms existing TextVQA methods, video-language pretraining methods and video large language models by great margins.
Abstract (translated)
基于视频文本的视觉问答（Video TextVQA）是一项实用任务，旨在通过共同推理给定视频中的文本和视觉信息来回答问题。受到图像领域TextVQA发展的启发，现有的Video TextVQA方法利用语言模型（例如T5）处理富含文本的多帧，并自回归地生成答案。然而，这种做法会破坏视觉实体（包括场景文本和对象）之间的时空关系，使模型容易受到无关信息的干扰，导致不合理推理和不准确的回答。为了解决这些问题，我们提出了TEA（意为“**T**rack th**E** **A**nswer”）方法，该方法更好地将生成性的TextVQA框架从图像扩展到视频。TEA以互补的方式恢复了时空关系，并融合了OCR感知线索来提高推理问题的质量。在多个公开的Video TextVQA数据集上进行的大量实验验证了我们框架的有效性和泛化能力。TEA显著优于现有的TextVQA方法、视频语言预训练方法和视频大型语言模型。
URL
https://arxiv.org/abs/2412.12502
