Abstract
Recent breakthroughs in Multimodal Large Language Models (MLLMs) have gained significant recognition within the deep learning community, where the fusion of the Video Foundation Models (VFMs) and Large Language Models(LLMs) has proven instrumental in constructing robust video understanding systems, effectively surmounting constraints associated with predefined visual tasks. These sophisticated MLLMs exhibit remarkable proficiency in comprehending videos, swiftly attaining unprecedented performance levels across diverse benchmarks. However, their operation demands substantial memory and computational resources, underscoring the continued importance of traditional models in video comprehension tasks. In this paper, we introduce a novel learning paradigm termed MLLM4WTAL. This paradigm harnesses the potential of MLLM to offer temporal action key semantics and complete semantic priors for conventional Weakly-supervised Temporal Action Localization (WTAL) methods. MLLM4WTAL facilitates the enhancement of WTAL by leveraging MLLM guidance. It achieves this by integrating two distinct modules: Key Semantic Matching (KSM) and Complete Semantic Reconstruction (CSR). These modules work in tandem to effectively address prevalent issues like incomplete and over-complete outcomes common in WTAL methods. Rigorous experiments are conducted to validate the efficacy of our proposed approach in augmenting the performance of various heterogeneous WTAL models.
Abstract (translated)
最近在多模态大型语言模型（MLLMs）方面的突破获得了深度学习社区的广泛认可，其中视频基础模型（VFMs）和大型语言模型（LLMs）的融合对于构建强大的视频理解系统至关重要，有效克服了预定义视觉任务相关的限制。这些高级的MLLM展示了对视频的理解能力，并在各种基准测试中迅速达到了前所未有的性能水平。然而，它们的操作需要大量的内存和计算资源，这突显了传统模型在视频理解任务中的持续重要性。本文介绍了一种名为MLLM4WTAL的新学习范式。该范式利用MLLM的潜力，为传统的弱监督时间动作定位（WTAL）方法提供时序动作关键语义和完整的语义先验。MLLM4WTAL通过利用MLLM指导来增强WTAL。它通过整合两个不同的模块——关键语义匹配（KSM）和完整语义重构（CSR）实现这一点。这两个模块协同工作，有效地解决了WTAL方法中常见的不完全和过度完成的问题。我们进行了严格的实验以验证我们的提议在提高各种异构WTAL模型性能方面的有效性。
URL
https://arxiv.org/abs/2411.08466
