Abstract
Despite decades of work, surveillance still struggles to find specific targets across long, multi-camera video. Prior methods -- tracking pipelines, CLIP based models, and VideoRAG -- require heavy manual filtering, capture only shallow attributes, and fail at temporal reasoning. Real-world searches are inherently multimodal (e.g., "When does this person join the fight?" with the person's image), yet this setting remains underexplored. Also, there are no proper benchmarks to evaluate those setting - asking video with multimodal queries. To address this gap, we introduce ForeSeaQA, a new benchmark specifically designed for video QA with image-and-text queries and timestamped annotations of key events. The dataset consists of long-horizon surveillance footage paired with diverse multimodal questions, enabling systematic evaluation of retrieval, temporal grounding, and multimodal reasoning in realistic forensic conditions. Not limited to this benchmark, we propose ForeSea, an AI forensic search system with a 3-stage, plug-and-play pipeline. (1) A tracking module filters irrelevant footage; (2) a multimodal embedding module indexes the remaining clips; and (3) during inference, the system retrieves top-K candidate clips for a Video Large Language Model (VideoLLM) to answer queries and localize events. On ForeSeaQA, ForeSea improves accuracy by 3.5% and temporal IoU by 11.0 over prior VideoRAG models. To our knowledge, ForeSeaQA is the first benchmark to support complex multimodal queries with precise temporal grounding, and ForeSea is the first VideoRAG system built to excel in this setting.
Abstract (translated)
尽管历经数十载研究，监控系统在长时段、多摄像头视频中定位特定目标仍面临挑战。先前的方案——包括追踪流水线、基于CLIP的模型及VideoRAG——均需大量人工筛选，仅能捕捉表层特征，且无法进行时序推理。现实中的搜索本质上是多模态的（例如：结合人物图像询问"此人何时加入斗殴？"），但此类场景尚未得到充分探索。此外，目前缺乏针对多模态查询视频检索场景的合适基准测试。为填补这一空白，我们推出了ForeSeaQA——首个专为图像-文本联合查询视频问答设计的新基准，包含带时间戳的关键事件标注。该数据集包含长时段监控视频与多样化多模态问题，可在真实取证条件下系统评估检索、时序定位与多模态推理能力。除基准外，我们还提出了ForeSea——一个三阶段即插即用式AI取证搜索系统：(1) 追踪模块过滤无关片段；(2) 多模态嵌入模块对剩余片段建立索引；(3) 推理阶段系统检索Top-K候选片段，交由视频大语言模型（VideoLLM）回答问题并定位事件。在ForeSeaQA上，ForeSea相比先前VideoRAG模型将准确率提升3.5%，时序交并比提升11.0。据我们所知，ForeSeaQA是首个支持复杂多模态查询与精确时序定位的基准，而ForeSea则是首个专为该场景优化的VideoRAG系统。
URL
https://arxiv.org/abs/2603.22872
