Abstract
Online video understanding often relies on individual frames, leading to frame-by-frame predictions. Recent advancements such as Online Temporal Action Localization (OnTAL), extend this approach to instance-level predictions. However, existing methods mainly focus on short-term context, neglecting historical information. To address this, we introduce the History-Augmented Anchor Transformer (HAT) Framework for OnTAL. By integrating historical context, our framework enhances the synergy between long-term and short-term information, improving the quality of anchor features crucial for classification and localization. We evaluate our model on both procedural egocentric (PREGO) datasets (EGTEA and EPIC) and standard non-PREGO OnTAL datasets (THUMOS and MUSES). Results show that our model outperforms state-of-the-art approaches significantly on PREGO datasets and achieves comparable or slightly superior performance on non-PREGO datasets, underscoring the importance of leveraging long-term history, especially in procedural and egocentric action scenarios. Code is available at: this https URL
Abstract (translated)
视频在线理解通常依赖于单个帧，导致帧对帧预测。 例如，在线时间动作定位 (OnTAL) 是一种 recent 进步，将此方法扩展到实例级别预测。然而，现有的方法主要关注短期上下文，忽视了历史信息。为了解决这个问题，我们引入了 OnTAL 的 History-Augmented Anchor Transformer (HAT) 框架。通过整合历史信息，我们的框架提高了长期和短期信息之间的协同作用，从而提高了分类和定位关键锚特征的质量。我们在 both 程序化自旋点 (PREGO) 数据集（EGTEA 和 EPIC）和非程序化 OnTAL 数据集（THUMOS 和 MUSES）上评估我们的模型。结果表明，我们的模型在 PREGO 数据集上显著优于最先进的 approaches，同时在非程序化 OnTAL 数据集上实现或稍微优于最先进的 approaches。这突出表明了利用长期历史在程序化和自旋点动作场景中的重要性。代码可在此处下载：https:// this URL
URL
https://arxiv.org/abs/2408.06437
