Video_Classification
-
Learning to Unify Deformable Shape and Texture Representations for Cardiac Video Classification
2026-07-08 15:15:16Tonmoy Hossain, Miaomiao ZhangAbstract
Deformable shape representations have proven to be robust complements to texture features in cardiac image classification, offering geometric priors that are invariant to imaging artifacts and intensity variations. However, existing deep networks perform simple concatenation to combine these distinct feature representations, which neither fully exploits their complementary nature nor learns cross-modal feature dependencies. Furthermore, this results in uniform attention across all timepoints; hence ignoring the varying diagnostic importance across the cardiac phases. In this paper, we propose a novel cardiac video classification model that, for the first time, learns temporal features in an integrated space of deformable shape and image texture representations. In particular, we design a bi-directional cross-attention in the latent space to fuse latent deformable shape and image features, allowing each modality to adaptively weight the other based on spatio-temporal correspondence. In contrast to current methods that apply uniform weighting across all the cardiac phases, our approach learns to dynamically adjust the contributions of shape and texture representations, derived from images, over time. We demonstrate state-of-the-art classification performance on a cine cardiac magnetic resonance (CMR) video dataset, achieving improved interpretability from attention mechanisms that identify diagnostically critical cardiac phases and modality contributions.
Abstract (translated)
URL
https://arxiv.org/abs/2607.07518
PDF
-
Gen4U: Unifying Video Generation and Understanding via Diffusion
2026-07-07 23:17:26Michael King, Aravindh Mahendran, Matthew Koichi Grimes, Fedor Kitashov, Adham Elarabawy, Pedro Velez, Maks Ovsjanikov, Viorica P\u{a}tr\u{a}uceanAbstract
Prior work suggests that diffusion representations capture low-level geometry but struggle with high-level semantics. We demonstrate that state-of-the-art video diffusion models overcome this limitation. By systematically probing their intermediate activations using recent mutual-kNN alignment metrics, we reveal a highly structured latent space where visual representations evolve across both network depth and noise levels. We show that while moderate noise levels yield linearly separable global semantics, fine-grained details persist at lower noise levels but become spatially scattered, requiring attention mechanisms to decode. Building on these insights, we introduce Gen4U (Generation for Understanding), a framework that repurposes these generative representations with a single forward pass. Our experiments establish that frozen, large-scale video diffusion models function as highly competitive video encoders across a wide spectrum of tasks, spanning semantic and non-semantic objectives (video classification, depth estimation, camera pose estimation, image and video captioning). Bypassing fine-tuning, Gen4U unifies the generation and understanding paradigms, achieving strong perception performance while fully preserving the model's ability to generate high-quality video.
Abstract (translated)
URL
https://arxiv.org/abs/2607.06856
PDF
-
Forget, Anticipate and Adapt: Test Time Training for Long Videos
2026-06-25 01:40:10Rajat Modi, Sebastian Noel, Xin Liang, Yogesh Singh RawatAbstract
Test Time Training (TTT) is a mechanism in which a model adapts to an incoming test-sample by performing some self-supervised (SSL) task and updating its weights even during inference. This procedure does not require labels at test-time. This paper focuses on TTT for long-videos. A major concern with existing approaches is: 1) they perform TTT updates using a sliding window containing frames in the past, whose compute increases linearly with the size of window. This becomes computationally intractable when the videos are hours long. 2) TTT is performed even when temporally close frames look similar, thereby consuming a lot of compute. We present the Frame Forgetting Network (FFN) that: 1) operates on only three frames within the sliding window, namely the frame that exits, the current frame and the frame after that. The model still manages to retain temporal context and work for hours long-videos; 2) mathematically define a surprise metric: how much new information the incoming frame contains with respect to the past seen frame. This facilitates determining how to modify the effective window size during TTT and constitutes the core mechanism of an adaptive windowing algorithm. Additionally, we curate a dataset EpicTours containing up to 3 hour long videos of walking city-tours, whereas earlier datasets on this problem were only 5 min long. We demonstrate FFNs empirical effectiveness on dense-segmentation, video classification tasks, generalization to depth-estimation, and multi-hour long videos.
Abstract (translated)
URL
https://arxiv.org/abs/2606.26515
PDF
-
Query-Efficient Video Adversarial Attack with Stylized Logo on Service Computing
2026-06-16 07:37:18Duoxun Tang, Yuxin Cao, Xi Xiao, Derui Wang, Sheng Wen, Tianqing ZhuAbstract
In service computing, video classification has become fundamental to many intelligent applications. While Deep Neural Networks (DNNs) have demonstrated excellent performance in recognizing video content, recent studies have shown that DNNs are highly vulnerable to adversarial examples. Thus, understanding adversarial attacks can better respond to emergency situations. In order to improve attack performance, many style-transfer-based attacks and patch-based attacks have been proposed. However, the global perturbation of the former will bring unnatural global colors, while the latter is difficult to achieve success in targeted attacks due to the limited perturbation space. Moreover, compared to a plethora of methods targeting image classifiers, video adversarial attacks remain relatively underexplored. Therefore, to generate adversarial examples with a low budget and to provide them with a higher verisimilitude, we propose a novel black-box video attack framework, called Stylized Logo Attack (SLA). SLA is conducted through three stages. The first stage involves building a style reference set for logos, which can not only make the generated examples more natural, but also carry more target class features in targeted attacks. Then, Reinforcement Learning is employed to determine the style reference and position parameters of the logo within the video, which ensures that the stylized logo is placed in the video with optimal attributes. Finally, perturbations are optimized in a step-by-step manner so as to improve the fooling rate. Experimental results indicate that SLA can achieve better performance than state-of-the-art methods and still maintain good deception effects when facing various defense methods. We believe SLA can raise awareness among the security community about the reliability and security of video classification systems and serve as a memorandum of possible attack methods.
Abstract (translated)
URL
https://arxiv.org/abs/2408.12099
PDF
-
Spatio-Temporal Fusion Model for Standard View Classification of Echocardiographic Videos
2026-06-16 02:37:43Bo Gou, Jicheng Zhang, Jianlong Xiong, Tao He, Bentian Liu, Hai Wu, Yijiao Wang, Yu Zhang, Yujia Yang, Yun Dai, Jian Liu, Jie WangAbstract
Automated classification of standard echocardiographic views is crucial for efficient clinical workflow but faces three main challenges. First, publicly available datasets are scarce and limited in scale and view coverage. Second, the performance of some modern video-level architectures for echocardiographic view classification remains underexplored. Third, some view categories exhibit highly similar spatial appearances, making single-frame features insufficient for discrimination, while heterogeneous frame quality complicates robust temporal information fusion. To address these challenges, we release the Echocardiographic Videos of Nine Views (EV9V) dataset, comprising 5,138 videos, 910,579 frames, and 9 standard views, which is, to the best of our knowledge, the largest publicly available echocardiography video dataset. Using EV9V, we systematically benchmark representative video classification architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformers. Furthermore, we propose a Spatio-Temporal Fusion Model (STFM), an efficient dual-stream CNN-LSTM (Long Short-Term Memory) framework that jointly captures spatial anatomical structures and temporal cardiac dynamics. The proposed framework leverages uncertainty-aware learning to preferentially sample representative video segments during training and evidence-based fusion during inference, improving robustness to variations in frame quality across echocardiographic videos. Extensive experiments demonstrate that our method achieves competitive performance across diverse video classification models, validating the effectiveness of uncertainty-aware spatio-temporal learning for echocardiographic view classification. The code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2606.17437
PDF
-
Interpretable Temporal Facial-Region Motion Analysis for In-the-Wild Parkinson's Disease Video Classification
2026-06-08 19:15:39Riyadh Almushrafy (Majmaah University, Saudi Arabia)Abstract
Reduced facial expressivity is a common motor manifestation of Parkinson's disease (PD), often described as hypomimia or facial bradykinesia. This paper examines whether temporal motion descriptors extracted from facial-region keypoints can support in-the-wild PD-related video classification on the YouTubePD benchmark. Each video is represented using geometric descriptors from 14 predefined facial regions. Static geometry, normalized geometry, velocity-based descriptors, relative-velocity descriptors, and a GRU sequence baseline are compared under the same binary classification protocol. To assess stability and interpretability, the study includes seed-robustness analysis, region-level ablation, and permutation importance. The best result is obtained with normalized velocity descriptors and a Random Forest classifier, reaching a balanced accuracy of 0.826 and an AUROC of 0.855 on the held-out test split. Across 10 random seeds, this representation remains stable, with balanced accuracy of 0.810 +/- 0.018 and AUROC of 0.855 +/- 0.005. Overall, the results suggest that normalized facial-region motion is a lightweight and interpretable representation for YouTubePD video classification. The study is framed as a benchmark-level analysis and does not claim clinical severity assessment or MDS-UPDRS facial-expression scoring.
Abstract (translated)
URL
https://arxiv.org/abs/2606.10088
PDF
-
Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos
2026-05-18 18:04:54Yuqi Tang, Yang Shi, Zhuoran Zhang, Qixun Wang, Xuehai Bai, Yue Ding, Ruizhe Chen, Bohan Zeng, Xinlong Chen, Xuanyu Zhu, Bozhou Li, Yuran Wang, Yifan Dai, Chengzhuo Tong, Xinyu Liu, Yiyan Ji, Yujie Wei, Yuhao Dong, Shilin Yan, Fengxiang Wang, Yi-Fan Zhang, Haotian Wang, Yuanxing Zhang, Pengfei WanAbstract
Recent video generative models have greatly improved the realism of AI-generated videos, yet their outputs still exhibit artifacts such as temporal inconsistencies, structural distortions, and semantic incoherence. While Multimodal Large Language Models (MLLMs) show strong visual understanding capabilities, their ability to perceive and reason about such artifacts remains unclear. Existing benchmarks often lack systematic evaluation of artifact-aware perception and fine-grained diagnostic reasoning, especially across diverse AI-generated video domains beyond photorealistic content. To address this gap, we introduce Artifact-Bench, a comprehensive benchmark for evaluating MLLMs on AI-generated video artifact detection and analysis. We first establish a three-level hierarchical taxonomy of realism artifacts, covering photorealistic, animated, and CG-style videos. Based on this taxonomy, Artifact-Bench defines three complementary tasks: real vs. AI-generated video classification, pairwise realism comparison, and fine-grained artifact identification. Experiments on 19 leading MLLMs reveal substantial limitations in artifact perception and reasoning, with many models approaching random or even below-random performance in challenging settings. We further observe significant misalignment between MLLM judgments and human perceptual preferences, highlighting their limited reliability as general evaluators for AI-generated video realism.
Abstract (translated)
URL
https://arxiv.org/abs/2605.18984
PDF
-
Appearance-free Action Recognition: Zero-shot Generalization in Humans and a Two-Pathway Model
2026-04-17 20:13:52Prerana Kumar (Hertie Institute, University of Tuebingen, IMPRS-IS), Martin A. Giese (Hertie Institute, University of Tuebingen)Abstract
Action recognition is a fundamental ability for social species. Yet, its underlying computations are not well understood. Classical psychophysical studies using simplified stimuli have shown that humans can perceive body motion even under degradation of relevant shape cues. Recent work using real-world action videos and their appearance-free counterparts (that preserve motion but lack static shape cues) included explicit training of humans and models on the appearance-free videos. Whether humans and vision models generalize in a zero-shot manner to appearance-free transformations of real-world action videos is not yet known. To measure this generalization in humans, we conducted a laboratory-based psychophysics experiment. 22 participants were trained to recognize five action categories using naturalistic videos (UCF5 dataset), and tested zero-shot on two types of appearance-free transformations: (i) dense-noise motion videos from an existing dataset (AFD5) and (ii) random-dot appearance-free videos. We find that participants recognize actions in both types of appearance-free videos well above chance, albeit with reduced accuracy compared to naturalistic videos. To model this behavior, we developed a two-pathway 3D CNN-based model combining an RGB (form) stream and an optical flow (motion) stream, including a coherence-gating mechanism inspired by Gestalt common-fate grouping. Our model generalizes to both appearance-free datasets and outperforms contemporary video classification models, narrowing the gap to human performance. We find that the motion pathway is critical for generalization to appearance-free videos, while the form pathway improves performance on naturalistic videos. Our findings highlight the importance of motion-based representations for generalization to appearance-free videos, and support the use of multi-stream architectures to model video-based action recognition.
Abstract (translated)
URL
https://arxiv.org/abs/2604.16675
PDF
-
LOLGORITHM: Funny Comment Generation Agent For Short Videos
2026-04-09 11:58:54Xuan Ouyang, Senan Wang, Bouzhou Wang, Siyuan Xiahou, Jinrong Zhou, Yuekang LiAbstract
Short-form video platforms have become central to multimedia information dissemination, where comments play a critical role in driving engagement, propagation, and algorithmic feedback. However, existing approaches -- including video summarization and live-streaming danmaku generation -- fail to produce authentic comments that conform to platform-specific cultural and linguistic norms. In this paper, we present LOLGORITHM, a novel modular multi-agent framework for stylized short-form video comment generation. LOLGORITHM supports six controllable comment styles and comprises three core modules: video content summarization, video classification, and comment generation with semantic retrieval and hot meme augmentation. We further construct a bilingual dataset of 3,267 videos and 16,335 comments spanning five high-engagement categories across YouTube and Douyin. Evaluation combining automatic scoring and large-scale human preference analysis demonstrates that LOLGORITHM consistently outperforms baseline methods, achieving human preference selection rates of 80.46\% on YouTube and 84.29\% on Douyin across 107 respondents. Ablation studies confirm that these gains are attributable to the framework architecture rather than the choice of backbone LLM, underscoring the robustness and generalizability of our approach.
Abstract (translated)
URL
https://arxiv.org/abs/2604.09729
PDF
-
vAccSOL: Efficient and Transparent AI Vision Offloading for Mobile Robots
2026-03-17 15:44:44Adam Zahir, Michele Gucciardom Falk Selker, Anastasios Nanos, Kostis Papazafeiropoulos, Carlos J. Bernardos, Nicolas Weber, Roberto GonzalezAbstract
Mobile robots are increasingly deployed for inspection, patrol, and search-and-rescue operations, relying on computer vision for perception, navigation, and autonomous decision-making. However, executing modern vision workloads onboard is challenging due to limited compute resources and strict energy constraints. While some platforms include embedded accelerators, these are typically tied to proprietary software stacks, leaving user-defined workloads to run on resource-constrained companion computers. We present vAccSOL, a framework for efficient and transparent execution of AI-based vision workloads across heterogeneous robotic and edge platforms. vAccSOL integrates two components: SOL, a neural network compiler that generates optimized inference libraries with minimal runtime dependencies, and vAccel, a lightweight execution framework that transparently dispatches inference locally on the robot or to nearby edge infrastructure. This combination enables hardware-optimized inference and flexible execution placement without requiring modifications to robot applications. We evaluate vAccSOL on a real-world testbed with a commercial quadruped robot and twelve deep learning models covering image classification, video classification, and semantic segmentation. Compared to a PyTorch compiler baseline, SOL achieves comparable or better inference performance. With edge offloading, vAccSOL reduces robot-side power consumption by up to 80% and edge-side power by up to 60% compared to PyTorch, while increasing vision pipeline frame rate by up to 24x, extending the operating lifetime of battery-powered robots.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16685
PDF
-
HSEmotion Team at ABAW-10 Competition: Facial Expression Recognition, Valence-Arousal Estimation, Action Unit Detection and Fine-Grained Violence Classification
2026-03-13 06:13:46Andrey V. Savchenko, Kseniia TsypliakovaAbstract
This article presents our results for the 10th Affective Behavior Analysis in-the-Wild (ABAW) competition. For frame-wise facial emotion understanding tasks (frame-wise facial expression recognition, valence-arousal estimation, action unit detection), we propose a fast approach based on facial embedding extraction with pre-trained EfficientNet-based emotion recognition models. If the latter model's confidence exceeds a threshold, its prediction is used. Otherwise, we feed embeddings into a simple multi-layered perceptron trained on the AffWild2 dataset. Estimated class-level scores are smoothed in a sliding window of fixed size to mitigate noise in frame-wise predictions. For the fine-grained violence detection task, we examine several pre-trained architectures for frame embeddings and their aggregation for video classification. Experimental results on four tasks from the ABAW challenge demonstrate that our approach significantly improves validation metrics over existing baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2603.12693
PDF
-
Contrastive learning-based video quality assessment-jointed video vision transformer for video recognition
2026-03-11 16:51:55Jian Sun, Mohammad H. MahoorAbstract
Video quality significantly affects video classification. We found this problem when we classified Mild Cognitive Impairment well from clear videos, but worse from blurred ones. From then, we realized that referring to Video Quality Assessment (VQA) may improve video classification. This paper proposed Self-Supervised Learning-based Video Vision Transformer combined with No-reference VQA for video classification (SSL-V3) to fulfill the goal. SSL-V3 leverages Combined-SSL mechanism to join VQA into video classification and address the label shortage of VQA, which commonly occurs in video datasets, making it impossible to provide an accurate Video Quality Score. In brief, Combined-SSL takes video quality score as a factor to directly tune the feature map of the video classification. Then, the score, as an intersected point, links VQA and classification, using the supervised classification task to tune the parameters of VQA. SSL-V3 achieved robust experimental results on two datasets. For example, it reached an accuracy of 94.87% on some interview videos in the I-CONECT (a facial video-involved healthcare dataset), verifying SSL-V3's effectiveness.
Abstract (translated)
视频质量显著影响视频分类的效果。我们在对轻度认知障碍（Mild Cognitive Impairment）进行分类时发现，从清晰的视频中可以较好地识别，但从模糊的视频中则表现较差。因此我们意识到借鉴视频质量评估（VQA）的方法可能有助于改进视频分类任务。本文提出了一种基于自监督学习的视频视觉变换器结合无参考VQA的视频分类方法（SSL-V3），以实现这一目标。 SSL-V3利用了Combined-SSL机制，将VQA与视频分类结合起来，并解决了VQA中常见的标签短缺问题，在大多数视频数据集中难以提供准确的视频质量分数。简而言之，Combined-SSL通过将视频质量评分作为一个因素直接调整视频分类模型中的特征图来实现这一点。同时，该评分作为交点位置，连接了VQA和分类任务，使用监督学习下的分类任务来调整VQA参数。 在两个数据集上，SSL-V3取得了稳健的实验结果。例如，在I-CONECT（一个涉及面部视频的医疗健康数据集）中的某些访谈视频上，其准确率达到了94.87%，证明了SSL-V3的有效性。
URL
https://arxiv.org/abs/2603.10965
PDF
-
From Imitation to Intuition: Intrinsic Reasoning for Open-Instance Video Classification
2026-03-11 00:50:01Ke Zhang, Xiangchen Zhao, Yunjie Tian, Jiayu Zheng, Vishal M. Patel, Di FuAbstract
Conventional video classification models, acting as effective imitators, excel in scenarios with homogeneous data distributions. However, real-world applications often present an open-instance challenge, where intra-class variations are vast and complex, beyond existing benchmarks. While traditional video encoder models struggle to fit these diverse distributions, vision-language models (VLMs) offer superior generalization but have not fully leveraged their reasoning capabilities (intuition) for such tasks. In this paper, we bridge this gap with an intrinsic reasoning framework that evolves open-instance video classification from imitation to intuition. Our approach, namely DeepIntuit, begins with a cold-start supervised alignment to initialize reasoning capability, followed by refinement using Group Relative Policy Optimization (GRPO) to enhance reasoning coherence through reinforcement learning. Crucially, to translate this reasoning into accurate classification, DeepIntuit then introduces an intuitive calibration stage. In this stage, a classifier is trained on this intrinsic reasoning traces generated by the refined VLM, ensuring stable knowledge transfer without distribution mismatch. Extensive experiments demonstrate that for open-instance video classification, DeepIntuit benefits significantly from transcending simple feature imitation and evolving toward intrinsic reasoning. Our project is available at this https URL.
Abstract (translated)
传统的视频分类模型在数据分布同质化的情景下表现出色，然而，在现实世界的应用中经常遇到开放实例挑战。在这种情况下，同一类别的内部变化是广泛且复杂的，超出了现有的基准测试范围。尽管传统视频编码器模型难以适应这些多样的分布情况，视觉-语言模型（VLMs）提供了更好的泛化能力，但它们尚未完全利用其推理能力来处理此类任务。本文提出了一种内在推理框架，旨在将开放实例视频分类从模仿进化到直觉理解。我们为此开发的方法被称为DeepIntuit。 DeepIntuit方法首先通过冷启动监督对齐初始化推理能力，然后使用组相对策略优化（GRPO）进行细化，以增强通过强化学习实现的推理一致性。至关重要的是，为了将这种推理转化为准确的分类，DeepIntuit引入了一个直观校准阶段，在这个阶段中，一个分类器会在经过改进的VLM生成的内在推理痕迹上进行训练，确保知识传递稳定且无分布错配。 广泛的实验表明，对于开放实例视频分类而言，DeepIntuit方法通过超越简单的特征模仿并进化到内在推理获益颇丰。我们的项目可在此链接访问：[此URL]（请将括号内的内容替换为实际的网址）。
URL
https://arxiv.org/abs/2603.10300
PDF
-
Assessing Situational and Spatial Awareness of VLMs with Synthetically Generated Video
2026-01-22 09:14:11Pascal Benschop, Justin Dauwels, Jan van GemertAbstract
Spatial reasoning in vision language models (VLMs) remains fragile when semantics hinge on subtle temporal or geometric cues. We introduce a synthetic benchmark that probes two complementary skills: situational awareness (recognizing whether an interaction is harmful or benign) and spatial awareness (tracking who does what to whom, and reasoning about relative positions and motion). Through minimal video pairs, we test three challenges: distinguishing violence from benign activity, binding assailant roles across viewpoints, and judging fine-grained trajectory alignment. While we evaluate recent VLMs in a training-free setting, the benchmark is applicable to any video classification model. Results show performance only slightly above chance across tasks. A simple aid, stable color cues, partly reduces assailant role confusions but does not resolve the underlying weakness. By releasing data and code, we aim to provide reproducible diagnostics and seed exploration of lightweight spatial priors to complement large-scale pretraining.
Abstract (translated)
在视觉语言模型（VLMs）中，空间推理依然脆弱，尤其是在语义依赖于微妙的时间或几何线索时。我们引入了一个合成基准测试，该测试考察两种互补技能：情境意识（识别互动是否具有危害性或无害）和空间意识（追踪谁对谁做了什么，并推断相对位置和运动）。通过最小化视频配对，我们测试了三个挑战：区分暴力行为与无害活动、在不同视角下绑定施暴者角色以及判断细微的轨迹对齐。尽管我们在零训练设置中评估了最近的VLMs，但该基准适用于任何视频分类模型。结果表明，在各个任务上的表现仅略高于随机猜测水平。一种简单的辅助手段——稳定颜色线索——部分减少了施暴者角色混淆，但仍未能解决根本弱点。通过发布数据和代码，我们旨在提供可重复诊断工具，并促进对轻量级空间先验的探索，以补充大规模预训练。
URL
https://arxiv.org/abs/2601.15780
PDF
-
Not all Blends are Equal: The BLEMORE Dataset of Blended Emotion Expressions with Relative Salience Annotations
2026-01-19 16:59:45Tim Lachmann, Alexandra Israelsson, Christina Tornberg, Teimuraz Saghinadze, Michal Balazia, Philipp M\"uller, Petri LaukkaAbstract
Humans often experience not just a single basic emotion at a time, but rather a blend of several emotions with varying salience. Despite the importance of such blended emotions, most video-based emotion recognition approaches are designed to recognize single emotions only. The few approaches that have attempted to recognize blended emotions typically cannot assess the relative salience of the emotions within a blend. This limitation largely stems from the lack of datasets containing a substantial number of blended emotion samples annotated with relative salience. To address this shortcoming, we introduce BLEMORE, a novel dataset for multimodal (video, audio) blended emotion recognition that includes information on the relative salience of each emotion within a blend. BLEMORE comprises over 3,000 clips from 58 actors, performing 6 basic emotions and 10 distinct blends, where each blend has 3 different salience configurations (50/50, 70/30, and 30/70). Using this dataset, we conduct extensive evaluations of state-of-the-art video classification approaches on two blended emotion prediction tasks: (1) predicting the presence of emotions in a given sample, and (2) predicting the relative salience of emotions in a blend. Our results show that unimodal classifiers achieve up to 29% presence accuracy and 13% salience accuracy on the validation set, while multimodal methods yield clear improvements, with ImageBind + WavLM reaching 35% presence accuracy and HiCMAE 18% salience accuracy. On the held-out test set, the best models achieve 33% presence accuracy (VideoMAEv2 + HuBERT) and 18% salience accuracy (HiCMAE). In sum, the BLEMORE dataset provides a valuable resource to advancing research on emotion recognition systems that account for the complexity and significance of blended emotion expressions.
Abstract (translated)
人类在某一时刻通常体验的不是单一的基本情绪，而是一种混合了多种不同强度情感的状态。尽管这种复合情感的重要性不容忽视，但大多数基于视频的情绪识别方法仅设计用于识别单一情绪。少数尝试识别复合情感的方法通常也无法评估这些情感之间的相对显著性。这一局限主要源于缺少包含大量注有相对显著性的复合情感样本的数据集。 为了解决这个问题，我们引入了BLEMORE数据集，这是一个新的多模态（视频、音频）混合情绪识别数据集，包含了每种情绪在混合中的相对显著信息。BLEMORE包括超过3000个片段，来自58名演员的表演，涵盖了6种基本情感和10种不同的复合情感组合，其中每一种复合情感有三种不同的情感强度配置（50/50, 70/30 和 30/70）。利用这个数据集，我们对最先进的视频分类方法进行了广泛的评估，针对两个混合情绪预测任务：(1) 预测给定样本中是否存在特定情绪；(2) 预测情感混合中的相对显著性。我们的结果显示，在验证集中，单模态分类器在存在准确性上可达到最高29%，在显著性准确性上可达13%；而多模态方法则明显表现出改进，其中ImageBind + WavLM达到了35%的存在准确性，HiCMAE的显著性准确率为18%。在未公开的测试集中，最佳模型实现了VideoMAEv2 + HuBERT的33%存在准确性和HiCMAE的18%显著性准确度。 综上所述，BLEMORE数据集为推进考虑情感表达复杂性和重要性的混合情绪识别系统研究提供了宝贵的资源。
URL
https://arxiv.org/abs/2601.13225
PDF
-
Effects of Different Attention Mechanisms Applied on 3D Models in Video Classification
2026-01-15 20:45:56Mohammad Rasras, Iuliana Marin, Serban Radu, Irina MocanuAbstract
Human action recognition has become an important research focus in computer vision due to the wide range of applications where it is used. 3D Resnet-based CNN models, particularly MC3, R3D, and R(2+1)D, have different convolutional filters to extract spatiotemporal features. This paper investigates the impact of reducing the captured knowledge from temporal data, while increasing the resolution of the frames. To establish this experiment, we created similar designs to the three originals, but with a dropout layer added before the final classifier. Secondly, we then developed ten new versions for each one of these three designs. The variants include special attention blocks within their architecture, such as convolutional block attention module (CBAM), temporal convolution networks (TCN), in addition to multi-headed and channel attention mechanisms. The purpose behind that is to observe the extent of the influence each of these blocks has on performance for the restricted-temporal models. The results of testing all the models on UCF101 have shown accuracy of 88.98% for the variant with multiheaded attention added to the modified R(2+1)D. This paper concludes the significance of missing temporal features in the performance of the newly created increased resolution models. The variants had different behavior on class-level accuracy, despite the similarity of their enhancements to the overall performance.
Abstract (translated)
人类行为识别由于其广泛的应用场景，已成为计算机视觉领域的重要研究焦点。基于3D Resnet的CNN模型（如MC3、R3D和R(2+1)D）通过不同的卷积滤波器来提取空间-时间特征。本文探讨了减少捕获的时间数据知识的影响，并增加帧分辨率的效果。为了进行这项实验，我们创建了与原始设计相似的设计，但在最终分类器之前添加了一个dropout层。其次，为每种设计方案开发了十个新版本，这些变体在其架构中加入了特殊的注意模块，如卷积块注意力模块（CBAM）、时间卷积网络（TCN），以及多头和通道注意机制。 引入这些模块的目的是观察它们在限制时间特征的新建高分辨率模型性能中的影响程度。所有模型在UCF101数据集上的测试结果表明，在修改后的R(2+1)D中添加了多头注意力机制的变体获得了88.98%的准确率。本文得出结论，缺失的时间特性对新创建的、增加分辨率的模型性能具有重要意义。尽管这些变体对其整体性能进行了类似的增强，但它们在类别级别的准确性上表现出不同的行为。
URL
https://arxiv.org/abs/2601.10854
PDF
-
Cell Behavior Video Classification Challenge, a benchmark for computer vision methods in time-lapse microscopy
2026-01-15 10:14:16Raffaella Fiamma Cabini, Deborah Barkauskas, Guangyu Chen, Zhi-Qi Cheng, David E Cicchetti, Judith Drazba, Rodrigo Fernandez-Gonzalez, Raymond Hawkins, Yujia Hu, Jyoti Kini, Charles LeWarne, Xufeng Lin, Sai Preethi Nakkina, John W Peterson, Koert Schreurs, Ayushi Singh, Kumaran Bala Kandan Viswanathan, Inge MN Wortel, Sanjian Zhang, Rolf Krause, Santiago Fernandez Gonzalez, Diego Ulisse PizzagalliAbstract
The classification of microscopy videos capturing complex cellular behaviors is crucial for understanding and quantifying the dynamics of biological processes over time. However, it remains a frontier in computer vision, requiring approaches that effectively model the shape and motion of objects without rigid boundaries, extract hierarchical spatiotemporal features from entire image sequences rather than static frames, and account for multiple objects within the field of view. To this end, we organized the Cell Behavior Video Classification Challenge (CBVCC), benchmarking 35 methods based on three approaches: classification of tracking-derived features, end-to-end deep learning architectures to directly learn spatiotemporal features from the entire video sequence without explicit cell tracking, or ensembling tracking-derived with image-derived features. We discuss the results achieved by the participants and compare the potential and limitations of each approach, serving as a basis to foster the development of computer vision methods for studying cellular dynamics.
Abstract (translated)
捕捉复杂细胞行为的显微视频分类对于理解并量化生物过程的时间动态至关重要。然而，这仍然是计算机视觉领域的一个前沿问题，需要能够有效建模没有刚性边界的对象形状和运动、从整个图像序列而非静态帧中提取层级时空特征，并考虑视野内多个对象的方法。为此，我们组织了细胞行为视频分类挑战赛（CBVCC），根据三种方法对35种方法进行了基准测试：基于追踪衍生特性的分类、端到端的深度学习架构直接从整个视频序列中学习时空特征而无需显式细胞跟踪，或者结合追踪衍生与图像衍生特性进行集成。我们讨论了参赛者所取得的结果，并比较了每种方法的潜力和局限性，旨在为研究细胞动力学的计算机视觉方法的发展奠定基础。
URL
https://arxiv.org/abs/2601.10250
PDF
-
VL-JEPA: Joint Embedding Predictive Architecture for Vision-language
2025-12-11 18:59:22Delong Chen, Mustafa Shukor, Theo Moutakanni, Willy Chung, Jade Yu, Tejaswi Kasarla, Allen Bolourchi, Yann LeCun, Pascale FungAbstract
We introduce VL-JEPA, a vision-language model built on a Joint Embedding Predictive Architecture (JEPA). Instead of autoregressively generating tokens as in classical VLMs, VL-JEPA predicts continuous embeddings of the target texts. By learning in an abstract representation space, the model focuses on task-relevant semantics while abstracting away surface-level linguistic variability. In a strictly controlled comparison against standard token-space VLM training with the same vision encoder and training data, VL-JEPA achieves stronger performance while having 50% fewer trainable parameters. At inference time, a lightweight text decoder is invoked only when needed to translate VL-JEPA predicted embeddings into text. We show that VL-JEPA natively supports selective decoding that reduces the number of decoding operations by 2.85x while maintaining similar performance compared to non-adaptive uniform decoding. Beyond generation, the VL-JEPA's embedding space naturally supports open-vocabulary classification, text-to-video retrieval, and discriminative VQA without any architecture modification. On eight video classification and eight video retrieval datasets, the average performance VL-JEPA surpasses that of CLIP, SigLIP2, and Perception Encoder. At the same time, the model achieves comparable performance as classical VLMs (InstructBLIP, QwenVL) on four VQA datasets: GQA, TallyQA, POPE and POPEv2, despite only having 1.6B parameters.
Abstract (translated)
我们介绍了一种基于联合嵌入预测架构（JEPA）的视觉-语言模型——VL-JEPA。不同于传统的视觉-语言模型（VLMs），在生成时逐个自回归地产生标记，VL-JEPA 预测目标文本的连续嵌入。通过在一个抽象表示空间中学习，该模型专注于任务相关的语义，并忽略表面级的语言变异性。在使用相同的视觉编码器和训练数据进行严格控制对比的情况下，与标准词元空间VLM训练相比，VL-JEPA 达到了更强的性能表现，同时其可训练参数减少了50%。推理时，当需要将VL-JEPA预测的嵌入转换为文本时才调用轻量级文本解码器。 我们展示了VL-JEPA原生支持选择性解码，在保持与非自适应均匀解码相似性能的同时，减少了解码操作次数2.85倍。除了生成任务外，VL-JEPA 的嵌入空间自然地支持开放词汇分类、文本到视频检索和判别式视觉问答（VQA），无需进行任何架构修改。 在八个视频分类数据集和八个视频检索数据集中，平均性能表明VL-JEPA超越了CLIP、SigLIP2和感知编码器。同时，在GQA、TallyQA、POPE 和 POPEv2这四个VQA 数据集上，尽管参数量仅为1.6B，其性能与经典的视觉-语言模型（如InstructBLIP, QwenVL）相当。
URL
https://arxiv.org/abs/2512.10942
PDF
-
OmniFD: A Unified Model for Versatile Face Forgery Detection
2025-11-30 22:36:42Haotian Liu, Haoyu Chen, Chenhui Pan, You Hu, Guoying Zhao, Xiaobai LiAbstract
Face forgery detection encompasses multiple critical tasks, including identifying forged images and videos and localizing manipulated regions and temporal segments. Current approaches typically employ task-specific models with independent architectures, leading to computational redundancy and ignoring potential correlations across related tasks. We introduce OmniFD, a unified framework that jointly addresses four core face forgery detection tasks within a single model, i.e., image and video classification, spatial localization, and temporal localization. Our architecture consists of three principal components: (1) a shared Swin Transformer encoder that extracts unified 4D spatiotemporal representations from both images and video inputs, (2) a cross-task interaction module with learnable queries that dynamically captures inter-task dependencies through attention-based reasoning, and (3) lightweight decoding heads that transform refined representations into corresponding predictions for all FFD tasks. Extensive experiments demonstrate OmniFD's advantage over task-specific models. Its unified design leverages multi-task learning to capture generalized representations across tasks, especially enabling fine-grained knowledge transfer that facilitates other tasks. For example, video classification accuracy improves by 4.63% when image data are incorporated. Furthermore, by unifying images, videos and the four tasks within one framework, OmniFD achieves superior performance across diverse benchmarks with high efficiency and scalability, e.g., reducing 63% model parameters and 50% training time. It establishes a practical and generalizable solution for comprehensive face forgery detection in real-world applications. The source code is made available at this https URL.
Abstract (translated)
面部伪造检测涵盖了多个关键任务，包括识别篡改的图像和视频以及定位被操纵的空间区域和时间片段。目前的方法通常采用特定于每个任务的独立架构模型，导致计算冗余并忽略相关任务之间的潜在关联性。我们引入了OmniFD，这是一个统一框架，在单一模型中共同解决了四个核心面部伪造检测任务，即图像分类、视频分类、空间定位以及时间定位。 该架构由三个主要组件组成： 1. **共享的Swin Transformer编码器**：从图像和视频输入中提取统一的4D时空表示。 2. **跨任务交互模块**：带有可学习查询的模块通过基于注意力机制的推理动态地捕捉任务间的相互依赖关系。 3. **轻量级解码头**：将细化后的表示转换为所有FFD（Face Forgery Detection）任务对应的预测。 广泛实验表明，OmniFD相对于特定于任务的模型具有优势。其统一的设计利用多任务学习来捕获跨任务的一般化表征，并特别支持细粒度的知识转移以帮助其他任务。例如，在引入图像数据的情况下，视频分类准确率提高了4.63%。 此外，通过将图像、视频和四个任务整合在一个框架内，OmniFD在多个基准测试上实现了高效且可扩展的卓越性能，例如减少了63%的模型参数量以及50%的训练时间。它为现实应用中的综合面部伪造检测提供了实际可行且通用的解决方案。 该研究的源代码可在上述提供的链接中获取。
URL
https://arxiv.org/abs/2512.01128
PDF
-
RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classification
2025-11-19 23:12:18Meilong Xu, Di Fu, Jiaxing Zhang, Gong Yu, Jiayu Zheng, Xiaoling Hu, Dongdi Zhao, Feiyang Li, Chao Chen, Yong CaoAbstract
Vision Language Models (VLMs) are becoming increasingly integral to multimedia understanding; however, they often struggle with domain-specific video classification tasks, particularly in cases with limited data. This stems from a critical \textit{rationale gap}, where sparse domain data is insufficient to bridge the semantic distance between complex spatio-temporal content and abstract classification labels. We propose a two-stage self-improvement paradigm to bridge this gap without new annotations. First, we prompt the VLMs to generate detailed textual rationales for each video, compelling them to articulate the domain-specific logic. The VLM is then fine-tuned on these self-generated rationales, utilizing this intermediate supervision to align its representations with the nuances of the target domain. Second, conventional supervised fine-tuning (SFT) is performed on the task labels, achieving markedly higher effectiveness as a result of the model's pre-acquired domain reasoning. Extensive experiments on diverse datasets demonstrate that our method significantly outperforms direct SFT, validating self-generated rationale as an effective, annotation-efficient paradigm for adapting VLMs to domain-specific video analysis.
Abstract (translated)
URL
https://arxiv.org/abs/2511.15923
PDF
