Abstract
In recent years, online lecture videos have become an increasingly popular resource for acquiring new knowledge. Systems capable of effectively understanding/indexing lecture videos are thus highly desirable, enabling downstream tasks like question answering to help users efficiently locate specific information within videos. This work proposes PreMind, a novel multi-agent multimodal framework that leverages various large models for advanced understanding/indexing of presentation-style videos. PreMind first segments videos into slide-presentation segments using a Vision-Language Model (VLM) to enhance modern shot-detection techniques. Each segment is then analyzed to generate multimodal indexes through three key steps: (1) extracting slide visual content, (2) transcribing speech narratives, and (3) consolidating these visual and speech contents into an integrated understanding. Three innovative mechanisms are also proposed to improve performance: leveraging prior lecture knowledge to refine visual understanding, detecting/correcting speech transcription errors using a VLM, and utilizing a critic agent for dynamic iterative self-reflection in vision analysis. Compared to traditional video indexing methods, PreMind captures rich, reliable multimodal information, allowing users to search for details like abbreviations shown only on slides. Systematic evaluations on the public LPM dataset and an internal enterprise dataset are conducted to validate PreMind's effectiveness, supported by detailed analyses.
Abstract (translated)
近年来，线上讲座视频已成为获取新知识的重要资源。能够有效理解/索引这类视频的系统因此变得十分关键，这有助于下游任务（如问答）帮助用户高效地在视频中定位特定信息。本工作提出了一种名为PreMind的新颖多代理跨模态框架，该框架利用各种大型模型来实现对演示风格视频的高级理解和索引。PreMind首先通过视觉-语言模型(Vision-Language Model, VLM)将视频分割成幻灯片展示片段，以增强现代镜头检测技术的效果。接着，在三个关键步骤中分析每个片段生成跨模态索引：(1) 提取幻灯片视觉内容；(2) 转录语音叙述；以及 (3) 将这些视觉和听觉内容整合成统一的理解。此外，还提出了三种创新机制以提高性能：利用先前的讲座知识来细化视觉理解、使用VLM检测/纠正语音转录错误，以及通过一个批评代理进行动态迭代自反省来改进视觉分析。相比传统的视频索引方法，PreMind能够捕捉到丰富且可靠的跨模态信息，使用户能够搜索到仅在幻灯片上展示的缩写等细节信息。本研究还通过对公开的LPM数据集和内部企业数据集进行系统的评估，验证了PreMind的有效性，并附有详细的分析结果。
URL
https://arxiv.org/abs/2503.00162
