Abstract
To address the zero-shot temporal action localization (ZSTAL) task, existing works develop models that are generalizable to detect and classify actions from unseen categories. They typically develop a category-agnostic action detector and combine it with the Contrastive Language-Image Pre-training (CLIP) model to solve ZSTAL. However, these methods suffer from incomplete action proposals generated for \textit{unseen} categories, since they follow a frame-level prediction paradigm and require hand-crafted post-processing to generate action proposals. To address this problem, in this work, we propose a novel model named Generalizable Action Proposal generator (GAP), which can interface seamlessly with CLIP and generate action proposals in a holistic way. Our GAP is built in a query-based architecture and trained with a proposal-level objective, enabling it to estimate proposal completeness and eliminate the hand-crafted post-processing. Based on this architecture, we propose an Action-aware Discrimination loss to enhance the category-agnostic dynamic information of actions. Besides, we introduce a Static-Dynamic Rectifying module that incorporates the generalizable static information from CLIP to refine the predicted proposals, which improves proposal completeness in a generalizable manner. Our experiments show that our GAP achieves state-of-the-art performance on two challenging ZSTAL benchmarks, i.e., Thumos14 and ActivityNet1.3. Specifically, our model obtains significant performance improvement over previous works on the two benchmarks, i.e., +3.2% and +3.4% average mAP, respectively.
Abstract (translated)
为解决零样本时序动作定位（ZSTAL）问题，现有研究开发了可适用于检测和分类未见类别的动作的模型。他们通常开发了一个类无关的动作检测器，并将其与对比性语言与图像预训练（CLIP）模型相结合来解决ZSTAL。然而，这些方法由于遵循帧级预测范式并需要手工后处理生成动作建议而产生了对未见类别的动作提议的缺失。为了解决这个问题，在本文中，我们提出了一个名为可扩展动作提议生成器（GAP）的新模型，该模型可以与CLIP无缝交互并以整体方式生成动作建议。我们的GAP基于查询架构构建，并使用建议级的目标进行训练，使其能够估计提议的完整性并消除手工后处理。基于这种架构，我们提出了一个动作感知差异损失，以增强动作的类无关动态信息。此外，我们还引入了一个静态动态校正模块，它从CLIP中包含了可扩展的静态信息，以改进预测建议，从而以一种通用的方式提高提议的完整性。我们的实验结果表明，我们的GAP在两个具有挑战性的ZSTAL基准上实现了最先进的性能，即Thumos14和ActivityNet1.3。具体来说，我们的模型在两个基准上的平均mAP均比以前的工作高出+3.2%和+3.4%。
URL
https://arxiv.org/abs/2408.13777
