Abstract
Temporal action localization (TAL) involves dual tasks to classify and localize actions within untrimmed videos. However, the two tasks often have conflicting requirements for features. Existing methods typically employ separate heads for classification and localization tasks but share the same input feature, leading to suboptimal performance. To address this issue, we propose a novel TAL method with Cross Layer Task Decoupling and Refinement (CLTDR). Based on the feature pyramid of video, CLTDR strategy integrates semantically strong features from higher pyramid layers and detailed boundary-aware boundary features from lower pyramid layers to effectively disentangle the action classification and localization tasks. Moreover, the multiple features from cross layers are also employed to refine and align the disentangled classification and regression results. At last, a lightweight Gated Multi-Granularity (GMG) module is proposed to comprehensively extract and aggregate video features at instant, local, and global temporal granularities. Benefiting from the CLTDR and GMG modules, our method achieves state-of-the-art performance on five challenging benchmarks: THUMOS14, MultiTHUMOS, EPIC-KITCHENS-100, ActivityNet-1.3, and HACS. Our code and pre-trained models are publicly available at: this https URL.
Abstract (translated)
时间动作定位（TAL）涉及对未剪辑视频中的动作进行分类和定位的双重任务。然而，这两个任务通常对特征有相互冲突的要求。现有的方法通常为分类和定位任务使用独立的头部，但共享相同的输入特征，导致性能次优。为了解决这一问题，我们提出了一种新的TAL方法，即跨层任务解耦与细化（CLTDR）。基于视频的特征金字塔，CLTDR策略整合了来自较高金字塔层级的语义强特征和来自较低金字塔层级的详细边界感知特征，有效地分离了动作分类和定位任务。此外，还利用跨层次的多个特征来细化并校准分离后的分类和回归结果。最后，提出了一种轻量级的门控多粒度（GMG）模块，以全面提取和聚合视频在瞬间、局部和全局时间粒度上的特征。得益于CLTDR和GMG模块，我们的方法在五个具有挑战性的基准测试中达到了最先进的性能：THUMOS14, MultiTHUMOS, EPIC-KITCHENS-100, ActivityNet-1.3 和 HACS。我们的代码和预训练模型可以在以下链接获取：此 https URL。
URL
https://arxiv.org/abs/2412.09202
