Abstract
We introduce the task of Audible Action Temporal Localization, which aims to identify the spatio-temporal coordinates of audible movements. Unlike conventional tasks such as action recognition and temporal action localization, which broadly analyze video content, our task focuses on the distinct kinematic dynamics of audible actions. It is based on the premise that key actions are driven by inflectional movements; for example, collisions that produce sound often involve abrupt changes in motion. To capture this, we propose $TA^{2}Net$, a novel architecture that estimates inflectional flow using the second derivative of motion to determine collision timings without relying on audio input. $TA^{2}Net$ also integrates a self-supervised spatial localization strategy during training, combining contrastive learning with spatial analysis. This dual design improves temporal localization accuracy and simultaneously identifies sound sources within video frames. To support this task, we introduce a new benchmark dataset, $Audible623$, derived from Kinetics and UCF101 by removing non-essential vocalization subsets. Extensive experiments confirm the effectiveness of our approach on $Audible623$ and show strong generalizability to other domains, such as repetitive counting and sound source localization. Code and dataset are available at this https URL.
Abstract (translated)
我们介绍了“可听动作时间定位”任务，其目标是识别出可发声的动作的时空坐标。不同于传统的动作识别和时间动作定位等广泛分析视频内容的任务，我们的任务专注于捕捉可发声动作独特的动力学动态。这一任务基于这样的前提：关键动作是由转折性运动驱动的；例如，产生声音的碰撞通常涉及突然的速度变化。为了捕捉这一点，我们提出了$TA^{2}Net$（即TAA-Net），这是一种新颖的架构，它使用运动的二阶导数来估算转折流，并据此确定碰撞时间点，而无需依赖音频输入。此外，$TA^{2}Net$在训练过程中整合了一种自我监督的空间定位策略，结合对比学习与空间分析以优化性能。 这种双管齐下的设计不仅提升了时间定位的准确性，还能够同时识别视频帧内的声源位置。为了支持这一任务，我们推出了一套新的基准数据集——$Audible623$（即AUDIBLE-623），该数据集是从Kinetics和UCF101中去除非必要音调子集后构建的。广泛的实验验证了我们在$Audible623$上的方法的有效性，并展示了其在其他领域的强大泛化能力，如重复计数和声源定位。代码及数据集可在以下网址获得：[请参见原文链接]。
URL
https://arxiv.org/abs/2506.13320
