Abstract
Traditional temporal action localization (TAL) methods rely on large amounts of detailed annotated data, whereas few-shot TAL reduces this dependence by using only a few training samples to identify unseen action categories. However, existing few-shot TAL methods typically focus solely on video-level information, neglecting textual information, which can provide valuable semantic support for the localization task. Therefore, we propose a new few-shot temporal action localization method by Chain-of-Thought textual reasoning to improve localization performance. Specifically, we design a novel few-shot learning framework that leverages textual semantic information to enhance the model's ability to capture action commonalities and variations, which includes a semantic-aware text-visual alignment module designed to align the query and support videos at different levels. Meanwhile, to better express the temporal dependencies and causal relationships between actions at the textual level to assist action localization, we design a Chain of Thought (CoT)-like reasoning method that progressively guides the Vision Language Model (VLM) and Large Language Model (LLM) to generate CoT-like text descriptions for videos. The generated texts can capture more variance of action than visual features. We conduct extensive experiments on the publicly available ActivityNet1.3 and THUMOS14 datasets. We introduce the first dataset named Human-related Anomaly Localization and explore the application of the TAL task in human anomaly detection. The experimental results demonstrate that our proposed method significantly outperforms existing methods in single-instance and multi-instance scenarios. We will release our code, data and benchmark.
Abstract (translated)
传统的时间动作定位（TAL）方法依赖大量的详细标注数据，而少样本时间动作定位则通过仅使用少量训练样本识别未见过的动作类别来减少对大量标注数据的依赖。然而，现有的少样本时间动作定位方法通常只关注视频级别的信息，忽视了文本信息，后者可以为定位任务提供有价值的语义支持。因此，我们提出了一种新的基于Chain-of-Thought（CoT）文本推理的时间动作定位方法，以提高定位性能。 具体来说，我们设计了一个新颖的少样本学习框架，利用文本语义信息增强模型捕捉动作共性和变异性的能力。该框架包括一个具有语义感知能力的文图对齐模块，用于在不同层面上对查询视频和支持视频进行对齐。同时，为了更好地表达动作之间的时序依赖关系和因果关系（从文本层面），我们设计了一种类似于CoT的推理方法，逐步引导视觉语言模型(VLM) 和大型语言模型(LLM) 生成类似CoT的文本描述来帮助行动定位任务。这种方法产生的文本比视觉特征更能捕捉到动作的变化。 我们在公开可用的ActivityNet1.3和THUMOS14数据集上进行了广泛的实验，并引入了一个名为“人类相关异常定位”的新数据集，以探索时间动作定位在人类行为异常检测中的应用。实验结果表明，在单实例和多实例场景下，我们提出的方法显著优于现有方法。 我们将公开我们的代码、数据和基准测试。
URL
https://arxiv.org/abs/2504.13460
