Action_Localization
-
OnPoint: Offline-to-Online Multi-Level Distillation for Point-Supervised Online Temporal Action Localization
2026-07-01 00:32:04Sakib Reza, Gauri Jagatap, Mohsen Moghaddam, Octavia Camps, Andrea FanelliAbstract
Temporal Action Localization (TAL) typically relies on segment annotations or offline access to full videos, limiting scalability and online use. We introduce Point-Supervised Online TAL (POTAL), which localizes actions in streaming videos using only one temporal point per instance. To solve POTAL, we propose OnPoint, an offline-to-online multi-level distillation framework that transfers knowledge from a point-supervised offline teacher to an online student via (i) pseudo-segment instance distillation, (ii) class-activation sequence distillation, and (iii) anticipatory window-level distillation. We further improve robustness by incorporating the original point labels into student training and by refining anchor decoding with actionness-guided attention calibration. Experiments on five datasets show OnPoint consistently outperforms strong baselines, establishing a solid foundation for POTAL.
Abstract (translated)
URL
https://arxiv.org/abs/2607.00289
PDF
-
AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding
2026-06-04 13:28:51Qize Yu, Jiadi You, Yuran Wang, Jiaqi Liang, Bowen Ping, Yang Tian, Yue Chen, Minghong Cai, Zeying Gong, Ruihai Wu, Yinchuan Li, Junwei Liang, Yingcong ChenAbstract
Vision-Language-Action (VLA) models leverage the rich world knowledge of pretrained vision-language models (VLMs) to enable instruction-following robotic manipulation. However, the structural mismatch between VLM semantic spaces and embodied control policies often hinders the learning of precise perception--action mappings. To address this challenge, we propose \textbf{AffordanceVLA}, a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation to establish a more precise and robust perception--action mapping. Specifically, we progressively model manipulation priors through three complementary components: 1) \textbf{Which2Act} for object-centric grounding via visual latent prediction to suppress distractions; 2) \textbf{Where2Act} for 2D interaction localization via affordance map estimation; and 3) \textbf{How2Act} for 3D geometric reasoning to guide manipulation policies. These affordance cues provide spatially grounded, semantically conditioned, and action-coupled intermediate representations, thereby naturally bridging vision, language and action. We integrate these modules into a Mixture-of-Transformer (MoT) architecture with specialized experts and train the model using a three-stage training strategy with a progressive data curriculum. To overcome the scarcity of dense affordance labels in robotic datasets, we also develop a robust automated data augmentation pipeline. Extensive experiments on simulation and real-world demonstrate that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
Abstract (translated)
URL
https://arxiv.org/abs/2606.06155
PDF
-
3rd Place at CVPR 2026 CASTLE Challenge: Agentic Multi-View Long-Context Video Understanding via Hierarchical Knowledge Graph Retrieval
2026-06-01 09:01:32Raghad Albusayes, Munirah AlyahyaAbstract
This paper presents our winning methodology for the CASTLE 2026 Challenge at the CVPR 2026 EgoVis Workshop, where our team secured third place globally. The challenge tasks participants with answering highly complex visual, spatiotemporal, and verbal questions, including visual counting, action localization, multi-view tracking and speaker temporal reasoning, within massive, multimodal video streams. The underlying dataset consists of over 600 hours synchronized footage captured by 15 ego and exo camera sources. To tackle the extreme scale and long-context demands of this environment, we introduce a training-free agentic framework optimized for long-form video understanding. Our framework introduces two core architectural components: i) a Video Knowledge Graph that maps static and dynamic entities, their temporal relationships, and intersecting events to enable multi-hop relational reasoning, and ii) an adaptive agentic workflow that resolves complex queries through a hierarchical retrieval and indexing. Empirical results demonstrate that our framework achieves high zero-shot reasoning accuracy on long-context multi-view streams. Our code will be released at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2606.01933
PDF
-
ConTrans: Learning Text-enhanced Local-global Temporal Representations for Zero-shot Temporal Action Localization
2026-05-29 00:34:17Kanchan Keisham, Thenukan Pathmanathan, Thangarajah AkilanAbstract
Zero-shot Temporal Action Localization (ZS-TAL) aims to detect and locate previously unseen actions in untrimmed videos. However, existing approaches primarily focus on modeling long-range contextual information, often neglecting the critical relative-offset-based local correlations between video frames. Furthermore, their performance is hindered by limited feature representation capabilities due to the shallow nature of their network architectures. In this paper, we address these limitations by introducing a novel local-global multi-scale feature representation module. We propose a novel multi-scale encoder architecture, termed ConTrans, that integrates convolutional (Conv) inductive biases with transformer Self-attention to jointly capture fine-grained local dependencies and long-range global context, leading to more comprehensive feature representations than existing methods. Experimental evaluations on the ActivityNet-1.3 and THUMOS14 datasets demonstrate that ConTrans significantly outperforms existing methods, establishing a new benchmark for ZS-TAL.
Abstract (translated)
URL
https://arxiv.org/abs/2605.30689
PDF
-
Masked Diffusion Vision-Language Models for Temporal Action Localization
2026-05-28 12:39:04Fengshun Wang, Zhengbo Zhang, Zhigang TuAbstract
Temporal action localization (TAL) requires recognizing the target event and localizing its start and end times precisely in untrimmed videos. Recent vision-language formulations improve semantic reasoning and support language-conditioned outputs, but their autoregressive decoders still generate tokens from left to right, preventing later semantic evidence from revising earlier timestamp predictions. We adapt masked diffusion vision-language models (MDVLMs) to TAL so that semantic tokens and boundary tokens remain editable throughout iterative denoising with bidirectional attention, allowing temporal boundaries and semantic content to be refined jointly. Direct adaptation, however, creates two TAL-specific mismatches: standard masked diffusion training corrupts all positions uniformly at random, but the time tokens are more reliable when enough semantic context is available; and token-level cross-entropy does not reflect temporal IoU. To address these mismatches, we introduce a Planned Training Objective that uses boundary-aware masking and step-weighted reconstruction to rehearse the late recovery of time tokens, together with a Step-Level IoU Reward that provides overlap-aware supervision during denoising. A standard sequence-level cross-entropy term provides the base reconstruction signal. Experiments on ActivityNet-RTL, ActivityNet-1.3, and THUMOS-14 show that MDVLM-TAL improves both temporal reasoning and boundary localization over autoregressive vision-language baselines, with especially strong gains under stricter temporal IoU criteria.
Abstract (translated)
URL
https://arxiv.org/abs/2605.29858
PDF
-
Decoupling Spatio-Temporal Adapter for Fine-Grained Badminton Action Localization
2026-05-22 08:21:26Tianyu Wang (School of Economics and Management, Beihang University, Beijing 100191, China), Junjie Wu (School of Economics and Management, Beihang University, Beijing 100191, China, Key Laboratory of Data Intelligence and Management, Beihang University, Ministry of Industry and Information Technology, Beijing 100191, China), Jingquan Gao (School of Economics and Management, Beihang University, Beijing 100191, China), Shishuo Li (School of Economics and Management, Beihang University, Beijing 100191, China)Abstract
Temporal Action Localization (TAL) has been extensively studied in generic video understanding, while fine-grained sports scenarios, such as professional badminton, remain underexplored due to their complex and subtle spatio-temporal dynamics. In this paper, we focus on fine-grained TAL in professional badminton videos and introduce a new benchmark dataset, Fine-Badminton, which consists of 31 matches with 29 fine-grained stroke categories, covering 2104 rallies and 27597 annotated actions. To effectively capture the intricate motion patterns in such scenarios, we propose a Decoupling Spatio-Temporal Adapter (DSTA), which enables efficient modeling of spatio-temporal features within a parameter-efficient framework. Specifically, DSTA decomposes motion representation into three parallel branches, capturing temporal dynamics as well as vertical and horizontal spatial variations. The design allows the model to better distinguish subtle differences among fine-grained actions. Extensive experiments on both the Fine-Badminton dataset and the ShuttleSet benchmark demonstrate that the proposed method achieves state-of-the-art performance while introducing only a marginal increase in computational and parameter cost. These results validate the effectiveness and efficiency of the proposed approach for fine-grained temporal action localization.
Abstract (translated)
URL
https://arxiv.org/abs/2605.23355
PDF
-
Zero-Shot Temporal Action Localization Through Textual Guidance
2026-05-21 09:05:27Benedetta Liberatori, Alessandro Conti, Lorenzo Vaquero, Paolo Rota, Yiming Wang, Elisa RicciAbstract
Zero-shot temporal action localization (ZS-TAL) consists of classifying and localizing actions in untrimmed videos, where action classes are unseen at training time. Existing work uses Vision and Language Models (VLMs), taking advantage of their strong zero-shot transfer capabilities. Yet, these models face evident challenges with fine-grained action classification, making it difficult to directly use them to distinguish between the presence and absence of an action. Most current methods for ZS-TAL address these challenges by training models on large-scale video datasets, which require annotated data and often result in limited generalization performance. Recently, approaches discarding the use of labeled data have emerged as an alternative. Following this direction, we propose a novel approach, ``Textual Guidance for finer localization of actions in videos'' (TEGU), that compensates for the lack of supervision from training data by exploiting rich textual information derived from large language models and structured text extracted from captions. This additional linguistic context can improve fine-grained discrimination by providing richer cues about fine-grained action differences within videos. We validate the effectiveness of the proposed method by conducting experiments on the THUMOS14 and the ActivityNet-v1.3 datasets. Our results show that, by exploiting rich textual information for improved action localization, TEGU outperforms state-of-the-art ZS-TAL approaches that do not involve training
Abstract (translated)
URL
https://arxiv.org/abs/2605.22201
PDF
-
OZ-TAL: Online Zero-Shot Temporal Action Localization
2026-05-11 04:34:32Chaolei Han, Hongsong Wang, Xin Gong, Jie GuiAbstract
Online Temporal Action Localization (On-TAL) aims to detect the occurrence time and category of actions in untrimmed streaming videos immediately upon their completion. Recent advancements in this field focus on developing more sophisticated frameworks, shifting from Online Action Detection (OAD)-based aggregation paradigm to instance-level understanding. However, existing approaches are typically trained on specific domains and often exhibit limited generalization capabilities when applied to arbitrary videos, particularly in the presence of previously unseen actions. In this paper, we introduce a new task called Online Zero-shot Temporal Action Localization (OZ-TAL), which aims to detect previously unseen actions in an online fashion. Furthermore, we propose a training-free framework that leverages off-the-shelf Vision-Language Models (VLMs) while introducing additional mechanisms to enhance visual representations and mitigate their inherent biases. We establish new benchmarks and representative baselines for OZ-TAL on THUMOS14 and ActivityNet-1.3, and extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches under both offline and online zero-shot settings.
Abstract (translated)
URL
https://arxiv.org/abs/2605.09976
PDF
-
From Frames to Events: Rethinking Evaluation in Human-Centric Video Anomaly Detection
2026-04-10 13:52:18Narges Rashvand, Shanle Yao, Armin Danesh Pazho, Babak Rahimi Ardabili, Hamed TabkhiAbstract
Pose-based Video Anomaly Detection (VAD) has gained significant attention for its privacy-preserving nature and robustness to environmental variations. However, traditional frame-level evaluations treat video as a collection of isolated frames, fundamentally misaligned with how anomalies manifest and are acted upon in the real world. In operational surveillance systems, what matters is not the flagging of individual frames, but the reliable detection, localization, and reporting of a coherent anomalous event, a contiguous temporal episode with an identifiable onset and duration. Frame-level metrics are blind to this distinction, and as a result, they systematically overestimate model performance for any deployment that requires actionable, event-level alerts. In this work, we propose a shift toward an event-centric perspective in VAD. We first audit widely used VAD benchmarks, including SHT[19], CHAD[6], NWPUC[4], and HuVAD[25], to characterize their event structure. We then introduce two strategies for temporal event localization: a score-refinement pipeline with hierarchical Gaussian smoothing and adaptive binarization, and an end-to-end Dual-Branch Model that directly generates event-level detections. Finally, we establish the first event-based evaluation standard for VAD by adapting Temporal Action Localization metrics, including tIoU-based event matching and multi-threshold F1 evaluation. Our results quantify a substantial performance gap: while all SoTA models achieve frame-level AUC-ROC exceeding 52% on the NWPUC[4], their event-level localization precision falls below 10% even at a minimal tIoU=0.2, with an average event-level F1 of only 0.11 across all thresholds. The code base for this work is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.09327
PDF
-
Learning from Synthetic Data via Provenance-Based Input Gradient Guidance
2026-04-03 10:28:58Koshiro Nagano, Ryo Fujii, Ryo Hachiuma, Fumiaki Sato, Taiki Sekii, Hideo SaitoAbstract
Learning methods using synthetic data have attracted attention as an effective approach for increasing the diversity of training data while reducing collection costs, thereby improving the robustness of model discrimination. However, many existing methods improve robustness only indirectly through the diversification of training samples and do not explicitly teach the model which regions in the input space truly contribute to discrimination; consequently, the model may learn spurious correlations caused by synthesis biases and artifacts. Motivated by this limitation, this paper proposes a learning framework that uses provenance information obtained during the training data synthesis process, indicating whether each region in the input space originates from the target object, as an auxiliary supervisory signal to promote the acquisition of representations focused on target regions. Specifically, input gradients are decomposed based on information about target and non-target regions during synthesis, and input gradient guidance is introduced to suppress gradients over non-target regions. This suppresses the model's reliance on non-target regions and directly promotes the learning of discriminative representations for target regions. Experiments demonstrate the effectiveness and generality of the proposed method across multiple tasks and modalities, including weakly supervised object localization, spatio-temporal action localization, and image classification.
Abstract (translated)
基于合成数据的学习方法因其能有效提升训练数据多样性、降低采集成本，进而增强模型判别鲁棒性而受到关注。然而，现有方法大多仅通过训练样本多样化间接提升鲁棒性，并未明确指导模型识别输入空间中真正贡献判别能力的区域；因此模型可能习得由合成偏差与伪影导致的虚假关联。针对这一局限，本文提出一种利用训练数据合成过程中获取的溯源信息作为辅助监督信号的学习框架，该信息指示输入空间中各区域是否源自目标对象，从而促进模型聚焦于目标区域的表征学习。具体而言，在合成过程中基于目标与非目标区域的信息对输入梯度进行分解，并引入输入梯度引导机制以抑制非目标区域的梯度响应。该方法可抑制模型对非目标区域的依赖，并直接促进目标区域判别性表征的学习。实验表明，所提方法在弱监督目标定位、时空动作定位及图像分类等多个任务与模态中均具有有效性与通用性。
URL
https://arxiv.org/abs/2604.02946
PDF
-
RatSeizure: A Benchmark and Saliency-Context Transformer for Rat Seizure Localization
2026-03-25 01:17:30Ting Yu Tsai, An Yu, Lucy Lee, Felix X. -F. Ye, Damian S. Shin, Tzu-Jen Kao, Xin Li, Ming-Ching ChangAbstract
Animal models, particularly rats, play a critical role in seizure research for studying epileptogenesis and treatment response. However, progress is limited by the lack of datasets with precise temporal annotations and standardized evaluation protocols. Existing animal behavior datasets often have limited accessibility, coarse labeling, and insufficient temporal localization of clinically meaningful events. To address these limitations, we introduce RatSeizure, the first publicly benchmark for fine-grained seizure behavior analysis. The dataset consists of recorded clips annotated with seizure-related action units and temporal boundaries, enabling both behavior classification and temporal localization. We further propose RaSeformer, a saliency-context Transformer for temporal action localization that highlights behavior-relevant context while suppressing redundant cues. Experiments on RatSeizure show that RaSeformer achieves strong performance and provides a competitive reference model for this challenging task. We also establish standardized dataset splits and evaluation protocols to support reproducible benchmarking.
Abstract (translated)
动物模型，尤其是大鼠，在癫痫发生及治疗反应的发作研究中发挥着关键作用。然而，由于缺乏具有精确时序标注和标准化评估协议的数据集，相关研究进展受限。现有动物行为数据集通常存在可访问性有限、标注粗糙以及临床有意义事件的时序定位不足等问题。为应对这些局限，我们推出了RatSeizure——首个用于细粒度发作行为分析的公开基准数据集。该数据集包含经过标注的视频片段，标注内容涉及发作相关动作单元及时序边界，从而支持行为分类与时序定位双重任务。此外，我们提出了RaSeformer，一种用于时序动作定位的显著性上下文Transformer模型，该模型能够突出与行为相关的上下文信息，同时抑制冗余线索。在RatSeizure上的实验表明，RaSeformer取得了优异性能，并为这一具有挑战性的任务提供了有竞争力的参考模型。我们还建立了标准化的数据集划分方案与评估协议，以支持可复现的基准测试。
URL
https://arxiv.org/abs/2603.26780
PDF
-
A Two-stage Transformer Framework for Temporal Localization of Distracted Driver Behaviors
2026-03-22 04:09:47Gia-Bao Doan, Nam-Khoa Huynh, Minh-Nhat-Huy Ho, Khanh-Thanh-Khoa Nguyen, Thanh-Hai LeAbstract
The identification of hazardous driving behaviors from in-cabin video streams is essential for enhancing road safety and supporting the detection of traffic violations and unsafe driver actions. However, current temporal action localization techniques often struggle to balance accuracy with computational efficiency. In this work, we develop and evaluate a temporal action localization framework tailored for driver monitoring scenarios, particularly suitable for periodic inspection settings such as transportation safety checkpoints or fleet management assessment systems. Our approach follows a two-stage pipeline that combines VideoMAE-based feature extraction with an Augmented Self-Mask Attention (AMA) detector, enhanced by a Spatial Pyramid Pooling-Fast (SPPF) module to capture multi-scale temporal features. Experimental results reveal a distinct trade-off between model capacity and efficiency. At the feature extraction stage, the ViT-Giant backbone delivers higher representations with 88.09% Top-1 test accuracy, while the ViT-based variant proves to be a practical alternative, achieving 82.55% accuracy with significantly lower computational fine-tuning costs (101.85 GFLOPs/segment compared to 1584.06 GFLOPs/segment for Giant). In the downstream localization task, the integration of SPPF consistently improves performance across all configurations. Notably, the ViT-Giant + SPPF model achieves a peak mAP of 92.67%, while the lightweight ViT-based configuration maintains robust results.
Abstract (translated)
基于车内视频流识别危险驾驶行为，对于提升道路安全以及辅助检测交通违规与不安全驾驶操作至关重要。然而，当前时序动作定位技术常难以兼顾准确性与计算效率。本研究开发并评估了一种专为驾驶员监控场景设计的时序动作定位框架，尤其适用于交通运输安全检查站或车队管理评估系统等周期性检查场景。该方法采用两阶段流程，结合基于VideoMAE的特征提取与增强自掩码注意力（AMA）检测器，并引入空间金字塔池化快速（SPPF）模块以捕获多尺度时序特征。实验结果表明，模型能力与效率之间存在明显权衡：在特征提取阶段，ViT-Giant主干网络以88.09%的Top-1测试准确率提供了更优表征，而基于ViT的变体则展现出实用性，以82.55%的准确率实现了显著更低的微调计算成本（每片段101.85 GFLOPs，而Giant为1584.06 GFLOPs）。在下游定位任务中，SPPF的集成持续提升了所有配置的性能。值得注意的是，ViT-Giant + SPPF模型达到了92.67%的峰值mAP，而轻量级基于ViT的配置也保持了稳健的结果。
URL
https://arxiv.org/abs/2603.21048
PDF
-
Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization
2026-02-05 14:46:21Yunchuan Ma, Laiyun Qing, Guorong Li, Yuqing Liu, Yuankai Qi, Qingming HuangAbstract
Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (\textit{i.e.}, labeling only a single frame per action instance) to train a model to effectively locate action instances within untrimmed videos. Most existing approaches design the task head of models with only a point-supervised snippet-level classification, without explicit modeling of understanding temporal relationships among frames of an action. However, understanding the temporal relationships of frames is crucial because it can help a model understand how an action is defined and therefore benefits localizing the full frames of an action. To this end, in this paper, we design a multi-task learning framework that fully utilizes point supervision to boost the model's temporal understanding capability for action localization. Specifically, we design three self-supervised temporal understanding tasks: (i) Action Completion, (ii) Action Order Understanding, and (iii) Action Regularity Understanding. These tasks help a model understand the temporal consistency of actions across videos. To the best of our knowledge, this is the first attempt to explicitly explore temporal consistency for point supervision action localization. Extensive experimental results on four benchmark datasets demonstrate the effectiveness of the proposed method compared to several state-of-the-art approaches.
Abstract (translated)
点监督时间动作定位（PTAL）采用轻量级的帧标注范式（即，仅对每个动作实例标记一个帧），来训练模型以有效地在未剪辑的视频中定位动作实例。大多数现有方法设计了只有基于点监督的片段级别分类的任务头，并没有明确地建模理解动作帧之间的时序关系。然而，理解帧之间的时序关系至关重要，因为它可以帮助模型理解动作是如何定义的，并且有助于定位整个动作帧。为此，在本文中，我们设计了一个多任务学习框架，该框架充分利用了点监督以增强模型的时间理解能力来进行动作定位。具体来说，我们设计了三个自我监督的时间理解任务：（i）动作完成，（ii）动作顺序理解，和（iii）动作规律性理解。这些任务帮助模型理解动作在不同视频中的时间一致性。据我们所知，这是首次明确探索点监督动作定位中的时间一致性的尝试。我们在四个基准数据集上进行了广泛的实验，结果表明与几种最先进的方法相比，本文提出的方法是有效的。
URL
https://arxiv.org/abs/2602.05718
PDF
-
WS-IMUBench: Can Weakly Supervised Methods from Audio, Image, and Video Be Adapted for IMU-based Temporal Action Localization?
2026-02-02 09:22:35Pei Li, Jiaxi Yin, Lei Ouyang, Shihan Pan, Ge Wang, Han Ding, Fei WangAbstract
IMU-based Human Activity Recognition (HAR) has enabled a wide range of ubiquitous computing applications, yet its dominant clip classification paradigm cannot capture the rich temporal structure of real-world behaviors. This motivates a shift toward IMU Temporal Action Localization (IMU-TAL), which predicts both action categories and their start/end times in continuous streams. However, current progress is strongly bottlenecked by the need for dense, frame-level boundary annotations, which are costly and difficult to scale. To address this bottleneck, we introduce WS-IMUBench, a systematic benchmark study of weakly supervised IMU-TAL (WS-IMU-TAL) under only sequence-level labels. Rather than proposing a new localization algorithm, we evaluate how well established weakly supervised localization paradigms from audio, image, and video transfer to IMU-TAL under only sequence-level labels. We benchmark seven representative weakly supervised methods on seven public IMU datasets, resulting in over 3,540 model training runs and 7,080 inference evaluations. Guided by three research questions on transferability, effectiveness, and insights, our findings show that (i) transfer is modality-dependent, with temporal-domain methods generally more stable than image-derived proposal-based approaches; (ii) weak supervision can be competitive on favorable datasets (e.g., with longer actions and higher-dimensional sensing); and (iii) dominant failure modes arise from short actions, temporal ambiguity, and proposal quality. Finally, we outline concrete directions for advancing WS-IMU-TAL (e.g., IMU-specific proposal generation, boundary-aware objectives, and stronger temporal reasoning). Beyond individual results, WS-IMUBench establishes a reproducible benchmarking template, datasets, protocols, and analyses, to accelerate community-wide progress toward scalable WS-IMU-TAL.
Abstract (translated)
URL
https://arxiv.org/abs/2602.01850
PDF
-
Boosting Point-supervised Temporal Action Localization via Text Refinement and Alignment
2026-02-01 14:35:46Yunchuan Ma, Laiyun Qing, Guorong Li, Yuqing Liu, Yuankai Qi, Qingming HuangAbstract
Recently, point-supervised temporal action localization has gained significant attention for its effective balance between labeling costs and localization accuracy. However, current methods only consider features from visual inputs, neglecting helpful semantic information from the text side. To address this issue, we propose a Text Refinement and Alignment (TRA) framework that effectively utilizes textual features from visual descriptions to complement the visual features as they are semantically rich. This is achieved by designing two new modules for the original point-supervised framework: a Point-based Text Refinement module (PTR) and a Point-based Multimodal Alignment module (PMA). Specifically, we first generate descriptions for video frames using a pre-trained multimodal model. Next, PTR refines the initial descriptions by leveraging point annotations together with multiple pre-trained models. PMA then projects all features into a unified semantic space and leverages a point-level multimodal feature contrastive learning to reduce the gap between visual and linguistic modalities. Last, the enhanced multi-modal features are fed into the action detector for precise localization. Extensive experimental results on five widely used benchmarks demonstrate the favorable performance of our proposed framework compared to several state-of-the-art methods. Moreover, our computational overhead analysis shows that the framework can run on a single 24 GB RTX 3090 GPU, indicating its practicality and scalability.
Abstract (translated)
URL
https://arxiv.org/abs/2602.01257
PDF
-
Towards Mitigating Modality Bias in Vision-Language Models for Temporal Action Localization
2026-01-28 22:03:46Jiaqi Li, Guangming Wang, Shuntian Zheng, Minzhe Ni, Xiaoman Lu, Guanghui Ye, Yu GuanAbstract
Temporal Action Localization (TAL) requires identifying both the boundaries and categories of actions in untrimmed videos. While vision-language models (VLMs) offer rich semantics to complement visual evidence, existing approaches tend to overemphasize linguistic priors at the expense of visual performance, leading to a pronounced modality bias. We propose ActionVLM, a vision-language aggregation framework that systematically mitigates modality bias in TAL. Our key insight is to preserve vision as the dominant signal while adaptively exploiting language only when beneficial. To this end, we introduce (i) a debiasing reweighting module that estimates the language advantage-the incremental benefit of language over vision-only predictions-and dynamically reweights language modality accordingly, and (ii) a residual aggregation strategy that treats language as a complementary refinement rather than the primary driver. This combination alleviates modality bias, reduces overconfidence from linguistic priors, and strengthens temporal reasoning. Experiments on THUMOS14 show that our model outperforms state-of-the-art by up to 3.2% mAP.
Abstract (translated)
URL
https://arxiv.org/abs/2601.21078
PDF
-
PyraTok: Language-Aligned Pyramidal Tokenizer for Video Understanding and Generation
2026-01-22 18:58:55Onkar Susladkar, Tushar Prakash, Adheesh Juvekar, Kiet A. Nguyen, Dong-Hwan Jang, Inderjit S Dhillon, Ismini LourentzouAbstract
Discrete video VAEs underpin modern text-to-video generation and video understanding systems, yet existing tokenizers typically learn visual codebooks at a single scale with limited vocabularies and shallow language supervision, leading to poor cross-modal alignment and zero-shot transfer. We introduce PyraTok, a language-aligned pyramidal tokenizer that learns semantically structured discrete latents across multiple spatiotemporal resolutions. PyraTok builds on a pretrained video VAE and a novel Language aligned Pyramidal Quantization (LaPQ) module that discretizes encoder features at several depths using a shared large binary codebook, yielding compact yet expressive video token sequences. To tightly couple visual tokens with language, PyraTok jointly optimizes multi-scale text-guided quantization and a global autoregressive objective over the token hierarchy. Across ten benchmarks, PyraTok delivers state-of-the-art (SOTA) video reconstruction, consistently improves text-to-video quality, and sets new SOTA zero-shot performance on video segmentation, temporal action localization, and video understanding, scaling robustly to up to 4K/8K resolutions.
Abstract (translated)
离散视频变分自编码器（VAEs）是现代文本到视频生成和视频理解系统的基础，然而现有的标记化方法通常在单一尺度上学习有限词汇量的视觉代码本，并且语言监督较浅层，导致跨模态对齐效果不佳及零样本迁移性能差。我们引入了PyraTok，这是一种与语言相匹配的金字塔式标记器，它能够在多个时空分辨率下学习语义结构化的离散潜在变量。PyraTok基于预训练的视频VAE以及一种新颖的语言一致金字塔量化（LaPQ）模块构建而成，该模块通过共享的大二进制代码本来自不同深度对编码特征进行离散化处理，生成紧凑且表达力强的视频标记序列。 为了将视觉标记与语言紧密耦合，PyraTok同时优化多尺度文本引导量化和整个令牌层次上的全局自回归目标。在十项基准测试中，PyraTok提供了最先进的（SOTA）视频重建效果，在文本到视频质量上持续改进，并在视频分割、时间动作定位以及视频理解的零样本性能方面设立新的SOTA标准，能够稳健地扩展至4K/8K分辨率。
URL
https://arxiv.org/abs/2601.16210
PDF
-
Skeleton-Snippet Contrastive Learning with Multiscale Feature Fusion for Action Localization
2025-12-18 13:15:52Qiushuo Cheng, Jingjing Liu, Catherine Morgan, Alan Whone, Majid MirmehdiAbstract
The self-supervised pretraining paradigm has achieved great success in learning 3D action representations for skeleton-based action recognition using contrastive learning. However, learning effective representations for skeleton-based temporal action localization remains challenging and underexplored. Unlike video-level {action} recognition, detecting action boundaries requires temporally sensitive features that capture subtle differences between adjacent frames where labels change. To this end, we formulate a snippet discrimination pretext task for self-supervised pretraining, which densely projects skeleton sequences into non-overlapping segments and promotes features that distinguish them across videos via contrastive learning. Additionally, we build on strong backbones of skeleton-based action recognition models by fusing intermediate features with a U-shaped module to enhance feature resolution for frame-level localization. Our approach consistently improves existing skeleton-based contrastive learning methods for action localization on BABEL across diverse subsets and evaluation protocols. We also achieve state-of-the-art transfer learning performance on PKUMMD with pretraining on NTU RGB+D and BABEL.
Abstract (translated)
自监督预训练范式在使用对比学习方法从基于骨架的动作识别中学习3D动作表示方面取得了巨大成功。然而，为基于骨架的时间动作定位学习有效的表示仍然具有挑战性且研究不足。与视频级动作识别不同，检测动作边界需要能够捕捉到标签变化前后相邻帧之间细微差别的时间敏感特征。为此，我们提出了一个片段区分的预训练任务，将骨架序列密集地投影到非重叠段中，并通过对比学习促进跨视频区分这些段落的特性。此外，我们在基于骨架的动作识别模型的强大骨干网络基础上融合了中间特征和U形模块以增强帧级定位的特征分辨率。我们的方法在BABEL数据集的不同子集和评估协议上持续改进现有的基于骨架的对比学习方法用于动作定位的表现。我们还在PKUMMD上实现了最先进的迁移学习性能，使用NTU RGB+D和BABEL进行预训练。
URL
https://arxiv.org/abs/2512.16504
PDF
-
See It Before You Grab It: Deep Learning-based Action Anticipation in Basketball
2025-12-17 12:39:31Arnau Barrera Roy, Albert Clap\'es SintesAbstract
Computer vision and video understanding have transformed sports analytics by enabling large-scale, automated analysis of game dynamics from broadcast footage. Despite significant advances in player and ball tracking, pose estimation, action localization, and automatic foul recognition, anticipating actions before they occur in sports videos has received comparatively little attention. This work introduces the task of action anticipation in basketball broadcast videos, focusing on predicting which team will gain possession of the ball following a shot attempt. To benchmark this task, a new self-curated dataset comprising 100,000 basketball video clips, over 300 hours of footage, and more than 2,000 manually annotated rebound events is presented. Comprehensive baseline results are reported using state-of-the-art action anticipation methods, representing the first application of deep learning techniques to basketball rebound prediction. Additionally, two complementary tasks, rebound classification and rebound spotting, are explored, demonstrating that this dataset supports a wide range of video understanding applications in basketball, for which no comparable datasets currently exist. Experimental results highlight both the feasibility and inherent challenges of anticipating rebounds, providing valuable insights into predictive modeling for dynamic multi-agent sports scenarios. By forecasting team possession before rebounds occur, this work enables applications in real-time automated broadcasting and post-game analysis tools to support decision-making.
Abstract (translated)
计算机视觉和视频理解技术通过从广播画面中进行大规模自动化分析，已经彻底改变了体育数据分析。尽管在球员追踪、姿态估计、动作定位以及自动犯规识别方面取得了重大进展，但在体育视频中预测即将发生的动作却受到了相对较少的关注。这项工作引入了篮球比赛广播视频中的动作预判任务，重点关注在投篮尝试后哪一队将获得球权的预测。为了评估这一任务，我们提出了一套新的自采集数据集，包括10万段篮球视频片段、超过300小时的影像资料以及2000多个手动标注的篮板事件。 研究报告了使用最先进的动作预判方法所得到的全面基线结果，这是首次将深度学习技术应用于预测篮球比赛中的篮板情况。此外，还探讨了两个互补的任务：篮板分类和篮板定位，这表明该数据集支持广泛的篮球视频理解应用，并且目前没有类似的现有数据集可以比较。 实验结果显示出了在体育比赛中预判篮板的可行性和固有挑战性，为多代理动态场景下的预测建模提供了宝贵的见解。通过提前预测在篮板发生前哪一队将获得球权，这项工作为实时自动化广播和赛后分析工具中的决策支持应用开辟了道路。
URL
https://arxiv.org/abs/2512.15386
PDF
-
Multi-task Learning with Extended Temporal Shift Module for Temporal Action Localization
2025-12-12 00:34:51Anh-Kiet Duong, Petra Gomez-Kr\"amerAbstract
We present our solution to the BinEgo-360 Challenge at ICCV 2025, which focuses on temporal action localization (TAL) in multi-perspective and multi-modal video settings. The challenge provides a dataset containing panoramic, third-person, and egocentric recordings, annotated with fine-grained action classes. Our approach is built on the Temporal Shift Module (TSM), which we extend to handle TAL by introducing a background class and classifying fixed-length non-overlapping intervals. We employ a multi-task learning framework that jointly optimizes for scene classification and TAL, leveraging contextual cues between actions and environments. Finally, we integrate multiple models through a weighted ensemble strategy, which improves robustness and consistency of predictions. Our method is ranked first in both the initial and extended rounds of the competition, demonstrating the effectiveness of combining multi-task learning, an efficient backbone, and ensemble learning for TAL.
Abstract (translated)
我们介绍了针对2025年ICCV BinEgo-360挑战赛的解决方案，该挑战专注于多视角和多模态视频环境下的时序动作定位（TAL）。此次比赛提供的数据集包括全景、第三人称和第一人称视角的记录，并详细标注了细粒度的动作类别。我们的方法基于时间偏移模块（Temporal Shift Module, TSM），并将其扩展以处理TAL，通过引入背景类以及对固定长度且不重叠的时间间隔进行分类来实现这一点。我们采用了一个多任务学习框架，在该框架中同时优化场景分类和TAL的性能，并利用动作与环境之间的上下文线索。最后，我们通过加权集成策略整合多个模型，这提升了预测结果的鲁棒性和一致性。 我们的方法在竞赛初始阶段以及扩展阶段均获得第一名，证明了将多任务学习、高效的骨干网络（backbone）及集成学习相结合，对于解决TAL问题的有效性。
URL
https://arxiv.org/abs/2512.11189
PDF
