Abstract
Weakly supervised temporal action localization (WTAL) aims to detect action instances in untrimmed videos using only video-level annotations. Since many existing works optimize WTAL models based on action classification labels, they encounter the task discrepancy problem (i.e., localization-by-classification). To tackle this issue, recent studies have attempted to utilize action category names as auxiliary semantic knowledge through vision-language pre-training (VLP). However, there are still areas where existing research falls short. Previous approaches primarily focused on leveraging textual information from language models but overlooked the alignment of dynamic human action and VLP knowledge in a joint space. Furthermore, the deterministic representation employed in previous studies struggles to capture fine-grained human motions. To address these problems, we propose a novel framework that aligns human action knowledge and VLP knowledge in a probabilistic embedding space. Moreover, we propose intra- and inter-distribution contrastive learning to enhance the probabilistic embedding space based on statistical similarities. Extensive experiments and ablation studies reveal that our method significantly outperforms all previous state-of-the-art methods. Code is available at this https URL.
Abstract (translated)
弱监督的时间动作定位（WTAL）旨在通过仅使用视频级别的注释来检测未剪辑视频中的动作实例。由于许多现有工作基于动作分类标签优化WTAL模型，因此它们遇到了任务差异问题（即通过分类来定位）。为解决这个问题，最近的研究试图通过视觉语言预训练（VLP）利用动作类别名称作为辅助语义知识。然而，目前的研究仍存在不足之处。之前的解决方案主要关注利用语言模型的文本信息，但忽视了动态人类动作与VLP知识在共同空间中的对齐。此外，之前使用的确定性表示方法很难捕捉到细粒度的人类运动。为了解决这些问题，我们提出了一个新框架，将人类动作知识和VLP知识在概率嵌入空间中对齐。此外，我们还提出了内部和分布式对比学习来增强基于统计相似性的概率嵌入空间。大量实验和消融实验揭示了我们的方法显著优于所有先前的最优方法。代码可在此处访问：https://www.xxx.com/。
URL
https://arxiv.org/abs/2408.05955
