Abstract
We introduce EgoTextVQA, a novel and rigorously constructed benchmark for egocentric QA assistance involving scene text. EgoTextVQA contains 1.5K ego-view videos and 7K scene-text aware questions that reflect real-user needs in outdoor driving and indoor house-keeping activities. The questions are designed to elicit identification and reasoning on scene text in an egocentric and dynamic environment. With EgoTextVQA, we comprehensively evaluate 10 prominent multimodal large language models. Currently, all models struggle, and the best results (Gemini 1.5 Pro) are around 33% accuracy, highlighting the severe deficiency of these techniques in egocentric QA assistance. Our further investigations suggest that precise temporal grounding and multi-frame reasoning, along with high resolution and auxiliary scene-text inputs, are key for better performance. With thorough analyses and heuristic suggestions, we hope EgoTextVQA can serve as a solid testbed for research in egocentric scene-text QA assistance.
Abstract (translated)
我们介绍了EgoTextVQA，这是一个新颖且严谨构建的基准测试平台，用于评估涉及场景文本的以自我为中心（egocentric）的问题回答辅助。EgoTextVQA包含1500个第一人称视角视频和7000个反映户外驾驶和室内家居管理活动中真实用户需求的场景文字感知问题。这些问题旨在激发在动态且以自我为中心环境中对场景文本的识别与推理能力。通过使用EgoTextVQA，我们全面评估了10种突出的多模态大型语言模型。当前，所有模型都面临挑战，其中表现最佳的是Gemini 1.5 Pro，其准确率为约33%，这表明这些技术在以自我为中心的问题回答辅助方面存在严重的不足之处。 进一步的研究表明，精确的时间定位、多帧推理以及高分辨率和场景文本输入的辅助是提高性能的关键因素。通过详细的分析和启发式建议，我们希望EgoTextVQA能成为研究以自我为中心场景文字问题回答辅助的坚实测试平台。
URL
https://arxiv.org/abs/2502.07411
