Abstract
Temporal Action Localization (TAL) is a critical task in video analysis, identifying precise start and end times of actions. Existing methods like CNNs, RNNs, GCNs, and Transformers have limitations in capturing long-range dependencies and temporal causality. To address these challenges, we propose a novel TAL architecture leveraging the Selective State Space Model (S6). Our approach integrates the Feature Aggregated Bi-S6 block, Dual Bi-S6 structure, and a recurrent mechanism to enhance temporal and channel-wise dependency modeling without increasing parameter complexity. Extensive experiments on benchmark datasets demonstrate state-of-the-art results with mAP scores of 74.2% on THUMOS-14, 42.9% on ActivityNet, 29.6% on FineAction, and 45.8% on HACS. Ablation studies validate our method's effectiveness, showing that the Dual structure in the Stem module and the recurrent mechanism outperform traditional approaches. Our findings demonstrate the potential of S6-based models in TAL tasks, paving the way for future research.
Abstract (translated)
Temporal Action Localization（TAL）是视频分析中一个关键的任务，用于识别动作的确切开始和结束时间。现有的方法如CNNs、RNNs、GCNs和Transformer在捕捉长距离依赖和时间因果关系方面具有局限性。为了应对这些挑战，我们提出了一种新型的TAL架构，利用了选择状态空间模型（S6）。我们的方法结合了特征聚合 bi-S6 块、双 bi-S6 结构以及一个循环机制，在不增加参数复杂性的情况下，增强了时间与通道之间的依赖建模。在基准数据集上的大量实验证明，在THUMOS-14上的mAP得分达到了74.2%，在ActivityNet上的mAP得分达到了42.9%，在FineAction上的mAP得分达到了29.6%，在HACS上的mAP得分达到了45.8%。消融研究证实了我们的方法的有效性，表明 stem 模块中的双结构以及循环机制超越了传统方法。我们的研究证明了基于S6的模型的潜力，为未来的研究奠定了基础。
URL
https://arxiv.org/abs/2407.13078
