Abstract
Long video understanding presents significant challenges for vision-language models due to extremely long context windows. Existing solutions relying on naive chunking strategies with retrieval-augmented generation, typically suffer from information fragmentation and a loss of global coherence. We present HAVEN, a unified framework for long-video understanding that enables coherent and comprehensive reasoning by integrating audiovisual entity cohesion and hierarchical video indexing with agentic search. First, we preserve semantic consistency by integrating entity-level representations across visual and auditory streams, while organizing content into a structured hierarchy spanning global summary, scene, segment, and entity levels. Then we employ an agentic search mechanism to enable dynamic retrieval and reasoning across these layers, facilitating coherent narrative reconstruction and fine-grained entity tracking. Extensive experiments demonstrate that our method achieves good temporal coherence, entity consistency, and retrieval efficiency, establishing a new state-of-the-art with an overall accuracy of 84.1% on LVBench. Notably, it achieves outstanding performance in the challenging reasoning category, reaching 80.1%. These results highlight the effectiveness of structured, multimodal reasoning for comprehensive and context-consistent understanding of long-form videos.
Abstract (translated)
长视频理解对视觉-语言模型构成了重大挑战，因为需要处理极长的上下文窗口。现有的解决方案通常依赖于简单的分块策略和检索增强生成方法，这会导致信息碎片化并丧失全局连贯性。我们提出了HAVEN框架，这是一种统一的方法来解决长视频理解问题，通过整合音视频实体一致性以及层级视频索引与代理搜索机制，实现连贯且全面的推理。 首先，HAVEN通过将视觉和听觉流中的实体级表示进行集成以保持语义的一致性，并按全局摘要、场景、片段和实体级别来组织内容，形成一个结构化的层次体系。然后，它采用一种代理搜索机制，能够在这些层级之间动态地检索并推理，从而实现连贯的叙事重构以及细粒度的实体跟踪。 广泛的实验表明，我们的方法在时间一致性、实体一致性和检索效率方面表现出色，在LVBench数据集上实现了84.1%的整体准确率，并且在具有挑战性的推理类别中达到了80.1%，确立了新的技术基准。这些结果突显出结构化和多模态推理对于理解长视频内容的全面性与上下文一致性至关重要。 总的来说，HAVEN框架展示了一种处理长视频理解的新方法，这种方法不仅提高了信息检索的有效性和连贯性，还提升了实体识别的准确性，在复杂的视听数据处理中展示了显著的优势。
URL
https://arxiv.org/abs/2601.13719
