Video_Classification
-
Discrepancy-Aware Attention Network for Enhanced Audio-Visual Zero-Shot Learning
2024-12-16 12:35:56RunLin Yu, Yipu Gong, Wenrui Li, Aiwen Sun, Mengren ZhengAbstract
Audio-visual Zero-Shot Learning (ZSL) has attracted significant attention for its ability to identify unseen classes and perform well in video classification tasks. However, modal imbalance in (G)ZSL leads to over-reliance on the optimal modality, reducing discriminative capabilities for unseen classes. Some studies have attempted to address this issue by modifying parameter gradients, but two challenges still remain: (a) Quality discrepancies, where modalities offer differing quantities and qualities of information for the same concept. (b) Content discrepancies, where sample contributions within a modality vary significantly. To address these challenges, we propose a Discrepancy-Aware Attention Network (DAAN) for Enhanced Audio-Visual ZSL. Our approach introduces a Quality-Discrepancy Mitigation Attention (QDMA) unit to minimize redundant information in the high-quality modality and a Contrastive Sample-level Gradient Modulation (CSGM) block to adjust gradient magnitudes and balance content discrepancies. We quantify modality contributions by integrating optimization and convergence rate for more precise gradient modulation in CSGM. Experiments demonstrates DAAN achieves state-of-the-art performance on benchmark datasets, with ablation studies validating the effectiveness of individual modules.
Abstract (translated)
音频视觉零样本学习（ZSL）因其能够识别未见过的类别并在视频分类任务中表现出色而受到了广泛关注。然而，(G)ZSL中的模态不平衡问题导致过度依赖最优模态，降低了对未见类别的辨别能力。一些研究试图通过修改参数梯度来解决这个问题，但仍存在两个挑战：(a) 质量差异，即对于同一概念，不同模态提供的信息数量和质量有所不同。(b) 内容差异，在同一个模态内样本贡献的变化很大。为了解决这些问题，我们提出了一种差异感知注意力网络（DAAN），用于增强音频视觉ZSL。我们的方法引入了一个质量差异缓解注意单元（QDMA）来减少高质量模态中的冗余信息，并且使用对比样本级梯度调节模块（CSGM）来调整梯度大小并平衡内容差异。我们在CSGM中通过整合优化和收敛率来量化模态贡献，实现更精确的梯度调节。实验表明，DAAN在基准数据集上实现了最先进的性能，消融研究验证了各个模块的有效性。
URL
https://arxiv.org/abs/2412.11715
PDF
-
Hybrid Spiking Neural Network -- Transformer Video Classification Model
2024-11-29 20:10:51Aaron BateniAbstract
In recent years, Spiking Neural Networks (SNNs) have gathered significant interest due to their temporal understanding capabilities. This work introduces, to the best of our knowledge, the first Cortical Column like hybrid architecture for the Time-Series Data Classification Task that leverages SNNs and is inspired by the brain structure, inspired from the previous hybrid models. We introduce several encoding methods to use with this model. Finally, we develop a procedure for training this network on the training dataset. As an effort to make using these models simpler, we make all the implementations available to the public.
Abstract (translated)
URL
https://arxiv.org/abs/2412.00237
PDF
-
Context-Aware Detection of Mixed Critical Events using Video Classification
2024-11-24 10:19:43Filza Akhlaq, Alina Arshad, Muhammad Yehya Hayati, Jawwad A. Shamsi, Muhammad Burhan KhanAbstract
Detecting mixed-critical events through computer vision is challenging due to the need for contextual understanding to assess event criticality accurately. Mixed critical events, such as fires of varying severity or traffic incidents, demand adaptable systems that can interpret context to trigger appropriate responses. This paper addresses these challenges by proposing a versatile detection system for smart city applications, offering a solution tested across traffic and fire detection scenarios. Our contributions include an analysis of detection requirements and the development of a system adaptable to diverse applications, advancing automated surveillance for smart cities.
Abstract (translated)
通过计算机视觉检测混合关键事件颇具挑战性，因为需要对情境进行理解以准确评估事件的严重程度。例如不同严重程度的火灾或交通事故等混合关键事件要求系统具有可适应性，能够解读情境并触发适当的响应。本文针对这些挑战提出了一个适用于智慧城市应用的多功能检测系统，并在交通和火灾检测场景中进行了测试。我们的贡献包括对检测需求的分析以及开发了一个可以应用于多种场景的系统，从而推动了智慧城市的自动化监控发展。
URL
https://arxiv.org/abs/2411.15773
PDF
-
AdaCM$^2$: On Understanding Extremely Long-Term Video with Adaptive Cross-Modality Memory Reduction
2024-11-19 18:04:13Yuanbin Man, Ying Huang, Chengming Zhang, Bingzhe Li, Wei Niu, Miao YinAbstract
The advancements in large language models (LLMs) have propelled the improvement of video understanding tasks by incorporating LLMs with visual models. However, most existing LLM-based models (e.g., VideoLLaMA, VideoChat) are constrained to processing short-duration videos. Recent attempts to understand long-term videos by extracting and compressing visual features into a fixed memory size. Nevertheless, those methods leverage only visual modality to merge video tokens and overlook the correlation between visual and textual queries, leading to difficulties in effectively handling complex question-answering tasks. To address the challenges of long videos and complex prompts, we propose AdaCM$^2$, which, for the first time, introduces an adaptive cross-modality memory reduction approach to video-text alignment in an auto-regressive manner on video streams. Our extensive experiments on various video understanding tasks, such as video captioning, video question answering, and video classification, demonstrate that AdaCM$^2$ achieves state-of-the-art performance across multiple datasets while significantly reducing memory usage. Notably, it achieves a 4.5% improvement across multiple tasks in the LVU dataset with a GPU memory consumption reduction of up to 65%.
Abstract (translated)
大型语言模型（LLMs）的进步通过将这些模型与视觉模型相结合，推动了视频理解任务的改进。然而，大多数现有的基于LLM的模型（例如VideoLLaMA、VideoChat）仅限于处理短时长视频。最近尝试通过提取和压缩视觉特征到固定内存大小来理解长期视频。尽管如此，这些方法只利用视觉模态来合并视频令牌，并忽视了视觉与文本查询之间的关联性，导致难以有效地处理复杂的问答任务。为了解决长时间视频和复杂提示的挑战，我们提出了AdaCM$^2$，该模型首次引入了一种自适应跨模态记忆缩减方法，在自动回归方式下应用于视频流中的视频-文本对齐。我们在各种视频理解任务上的广泛实验，如视频字幕生成、视频问答和视频分类，表明AdaCM$^2$在多个数据集上达到了最先进的性能，并显著减少了内存使用量。值得注意的是，在LVU数据集中，它实现了跨多项任务4.5%的提升，同时GPU内存消耗减少高达65%。
URL
https://arxiv.org/abs/2411.12593
PDF
-
Efficient Audio-Visual Fusion for Video Classification
2024-11-08 14:47:28Mahrukh Awan, Asmar Nadeem, Armin MustafaAbstract
We present Attend-Fusion, a novel and efficient approach for audio-visual fusion in video classification tasks. Our method addresses the challenge of exploiting both audio and visual modalities while maintaining a compact model architecture. Through extensive experiments on the YouTube-8M dataset, we demonstrate that our Attend-Fusion achieves competitive performance with significantly reduced model complexity compared to larger baseline models.
Abstract (translated)
我们提出了Attend-Fusion，这是一种新颖且高效的视频分类任务中音视频融合方法。我们的方法解决了在保持紧凑模型架构的同时利用音频和视觉模态的挑战。通过在YouTube-8M数据集上的广泛实验，我们证明了与更大的基线模型相比，我们的Attend-Fusion在显著降低模型复杂度的情况下仍能实现具有竞争力的性能。
URL
https://arxiv.org/abs/2411.05603
PDF
-
Optimizing Violence Detection in Video Classification Accuracy through 3D Convolutional Neural Networks
2024-11-02 19:29:01Aarjav Kavathia, Simeon SayerAbstract
As violent crimes continue to happen, it becomes necessary to have security cameras that can rapidly identify moments of violence with excellent accuracy. The purpose of this study is to identify how many frames should be analyzed at a time in order to optimize a violence detection model's accuracy as a parameter of the depth of a 3D convolutional network. Previous violence classification models have been created, but their application to live footage may be flawed. In this project, a convolutional neural network was created to analyze optical flow frames of each video. The number of frames analyzed at a time would vary with one, two, three, ten, and twenty frames, and each model would be trained for 20 epochs. The greatest validation accuracy was 94.87% and occurred with the model that analyzed three frames at a time. This means that machine learning models to detect violence may function better when analyzing three frames at a time for this dataset. The methodology used to identify the optimal number of frames to analyze at a time could be used in other applications of video classification, especially those of complex or abstract actions, such as violence.
Abstract (translated)
随着暴力犯罪的持续发生，需要有能够快速且准确识别暴力时刻的安全摄像头。本研究旨在确定为了优化暴力检测模型的准确性，三维卷积网络深度参数下应该一次性分析多少帧。虽然已有一些暴力分类模型被创建出来，但它们在实时视频中的应用可能存在缺陷。在这个项目中，我们建立了一个卷积神经网络来分析每个视频的光流帧。一次分析的帧数分别为一、二、三、十和二十帧，并且每种模型都进行了20个周期的训练。最高的验证准确率达到了94.87%，这发生在一次性分析三帧的模型上。这意味着，对于这个数据集来说，机器学习模型在检测暴力时可能在一次分析三帧的情况下效果更好。用于确定最佳分析帧数的方法也可以应用于其他视频分类领域，特别是那些复杂或抽象动作的分类，如暴力行为识别。
URL
https://arxiv.org/abs/2411.01348
PDF
-
Video Token Merging for Long-form Video Understanding
2024-10-31 09:55:32Seon-Ho Lee, Jue Wang, Zhikang Zhang, David Fan, Xinyu LiAbstract
As the scale of data and models for video understanding rapidly expand, handling long-form video input in transformer-based models presents a practical challenge. Rather than resorting to input sampling or token dropping, which may result in information loss, token merging shows promising results when used in collaboration with transformers. However, the application of token merging for long-form video processing is not trivial. We begin with the premise that token merging should not rely solely on the similarity of video tokens; the saliency of tokens should also be considered. To address this, we explore various video token merging strategies for long-form video classification, starting with a simple extension of image token merging, moving to region-concentrated merging, and finally proposing a learnable video token merging (VTM) algorithm that dynamically merges tokens based on their saliency. Extensive experimental results show that we achieve better or comparable performances on the LVU, COIN, and Breakfast datasets. Moreover, our approach significantly reduces memory costs by 84% and boosts throughput by approximately 6.89 times compared to baseline algorithms.
Abstract (translated)
随着视频理解中数据和模型规模的迅速扩大，基于变压器模型处理长格式视频输入变得具有实际挑战性。与其依赖可能导致信息丢失的输入采样或标记丢弃，与变压器协作使用的标记合并显示出有希望的结果。然而，对长格式视频处理应用标记合并并不简单。我们从这样一个前提开始：标记合并不应仅依靠视频标记之间的相似性；还应考虑标记的重要性。为了解决这一问题，我们探索了多种针对长格式视频分类的视频标记合并策略，从图像标记合并的简单扩展入手，转向集中于区域的合并方法，最终提出了一种基于可学习的视频标记合并（VTM）算法，该算法根据标记的重要性动态地进行合并。广泛的实验结果表明，在LVU、COIN和Breakfast数据集上我们取得了更好的或相当的表现。此外，与基线算法相比，我们的方法显著降低了84%的内存成本，并将吞吐量提高了大约6.89倍。
URL
https://arxiv.org/abs/2410.23782
PDF
-
CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification
2024-10-03 22:58:56Jinghao Shi, Xiang Shen, Kaili Zhao, Xuedong Wang, Vera Wen, Zixuan Wang, Yifan Wu, Zhixin ZhangAbstract
Dense features, customized for different business scenarios, are essential in short video classification. However, their complexity, specific adaptation requirements, and high computational costs make them resource-intensive and less accessible during online inference. Consequently, these dense features are categorized as `Privileged Dense Features'.Meanwhile, end-to-end multi-modal models have shown promising results in numerous computer vision tasks. In industrial applications, prioritizing end-to-end multi-modal features, can enhance efficiency but often leads to the loss of valuable information from historical privileged dense this http URL integrate both features while maintaining efficiency and manageable resource costs, we present Confidence-aware Privileged Feature Distillation (CPFD), which empowers features of an end-to-end multi-modal model by adaptively distilling privileged features during this http URL existing privileged feature distillation (PFD) methods, which apply uniform weights to all instances during distillation, potentially causing unstable performance across different business scenarios and a notable performance gap between teacher model (Dense Feature enhanced multimodal-model DF-X-VLM) and student model (multimodal-model only X-VLM), our CPFD leverages confidence scores derived from the teacher model to adaptively mitigate the performance variance with the student this http URL conducted extensive offline experiments on five diverse tasks demonstrating that CPFD improves the video classification F1 score by 6.76% compared with end-to-end multimodal-model (X-VLM) and by 2.31% with vanilla PFD on-average. And it reduces the performance gap by 84.6% and achieves results comparable to teacher model DF-X-VLM. The effectiveness of CPFD is further substantiated by online experiments, and our framework has been deployed in production systems for over a dozen models.
Abstract (translated)
密集特征，针对不同的业务场景进行定制，是短时视频分类的基本要求。然而，它们的复杂性、特定的适应要求和高的计算成本使它们变得资源密集且难以在线推理时使用。因此，这些密集特征被归类为“特权密集特征”。与此同时，端到端多模态模型已经在许多计算机视觉任务中取得了良好的结果。在工业应用中，优先考虑端到端多模态特征，可以提高效率，但往往会导致历史特权密集特征的宝贵信息丢失。为了同时保持效率和可管理资源成本，我们提出了Confidence-aware Privileged Feature Distillation（CPFD），它通过在现有的特权特征提取方法中自适应地蒸馏特权特征，使端到端多模态模型的特征具有可塑性。通过应用教师模型的置信分数，CPFD可以动态地减轻学生模型的性能方差。我们在五个多样化的任务上进行了广泛的离线实验，结果表明，与端到端多模态模型（X-VLM）相比，CPFD可以将视频分类F1得分提高6.76%；与普通PFD相比，CPFD可以将性能差距减少84.6%。在实际应用中，CPFD的效果得到了进一步的验证，我们的框架已经被部署在多个生产系统上，支持了数十个模型。
URL
https://arxiv.org/abs/2410.03038
PDF
-
TikGuard: A Deep Learning Transformer-Based Solution for Detecting Unsuitable TikTok Content for Kids
2024-10-01 05:00:05Mazen Balat, Mahmoud Essam Gabr, Hend Bakr, Ahmed B. ZakyAbstract
The rise of short-form videos on platforms like TikTok has brought new challenges in safeguarding young viewers from inappropriate content. Traditional moderation methods often fall short in handling the vast and rapidly changing landscape of user-generated videos, increasing the risk of children encountering harmful material. This paper introduces TikGuard, a transformer-based deep learning approach aimed at detecting and flagging content unsuitable for children on TikTok. By using a specially curated dataset, TikHarm, and leveraging advanced video classification techniques, TikGuard achieves an accuracy of 86.7%, showing a notable improvement over existing methods in similar contexts. While direct comparisons are limited by the uniqueness of the TikHarm dataset, TikGuard's performance highlights its potential in enhancing content moderation, contributing to a safer online experience for minors. This study underscores the effectiveness of transformer models in video classification and sets a foundation for future research in this area.
Abstract (translated)
短形式视频在像TikTok这样的平台上崛起，为保护年轻观众免受不良内容带来了新的挑战。传统的审核方法通常在处理用户生成视频的广泛且快速变化的地形方面存在缺陷，增加了孩子们接触有害材料的风险。本文介绍了TikGuard，一种基于Transformer的深度学习方法，旨在检测和标记TikTok上不适合儿童的内容。通过使用专门挑选的数据集TikHarm和利用先进的视频分类技术，TikGuard实现了86.7%的准确率，表明其在类似环境中与现有方法相比取得了显著的改善。尽管直接比较受到TikHarm数据集独特性的限制，但TikGuard的表现突出了其在增强内容审核方面的潜力，为未成年人提供更安全在线体验。本研究突出了Transformer模型在视频分类中的有效性，为该领域未来的研究奠定了基础。
URL
https://arxiv.org/abs/2410.00403
PDF
-
Benchmarking Edge AI Platforms for High-Performance ML Inference
2024-09-23 08:27:27Rakshith Jayanth, Neelesh Gupta, Viktor PrasannaAbstract
Edge computing's growing prominence, due to its ability to reduce communication latency and enable real-time processing, is promoting the rise of high-performance, heterogeneous System-on-Chip solutions. While current approaches often involve scaling down modern hardware, the performance characteristics of neural network workloads on these platforms can vary significantly, especially when it comes to parallel processing, which is a critical consideration for edge deployments. To address this, we conduct a comprehensive study comparing the latency and throughput of various linear algebra and neural network inference tasks across CPU-only, CPU/GPU, and CPU/NPU integrated solutions. {We find that the Neural Processing Unit (NPU) excels in matrix-vector multiplication (58.6% faster) and some neural network tasks (3.2$\times$ faster for video classification and large language models). GPU outperforms in matrix multiplication (22.6% faster) and LSTM networks (2.7$\times$ faster) while CPU excels at less parallel operations like dot product. NPU-based inference offers a balance of latency and throughput at lower power consumption. GPU-based inference, though more energy-intensive, performs best with large dimensions and batch sizes. We highlight the potential of heterogeneous computing solutions for edge AI, where diverse compute units can be strategically leveraged to boost accurate and real-time inference.
Abstract (translated)
由于其能够降低通信延迟并实现实时处理，使边缘计算的重要性不断增加，推动了高性能、异构 System-on-Chip 解决方案的发展。然而，通常的方法往往会缩小现代硬件的规模，这些平台上的神经网络负载的性能特征差异很大，特别是在并行处理方面，这是边缘部署的关键考虑因素。为了应对这个问题，我们进行了全面的研究，比较了各种线性代数和神经网络推理任务在CPU-only、CPU/GPU和CPU/NPU集成解决方案中的延迟和吞吐量。我们发现，神经处理单元（NPU）在矩阵向量乘法（58.6% faster）和某些神经网络任务（视频分类和大语言模型的速度快3.2倍）方面表现优异。GPU 在矩阵乘法（22.6% faster）和LSTM网络（2.7倍 faster）方面优于CPU，而在并行操作方面，比如点积，CPU的性能较低。基于NPU的推理在较低的功耗下实现了平衡的延迟和吞吐量。尽管GPU 的性能更耗费能量，但在大维度和高批量的场景下表现最佳。我们强调了异构计算解决方案在边缘 AI 中的潜力，通过 strategic leveraging diverse compute units，可以提高准确和实时的推理。
URL
https://arxiv.org/abs/2409.14803
PDF
-
Pushing the boundaries of event subsampling in event-based video classification using CNNs
2024-09-13 16:14:45Hesam Araghi, Jan van Gemert, Nergis TomenAbstract
Event cameras offer low-power visual sensing capabilities ideal for edge-device applications. However, their high event rate, driven by high temporal details, can be restrictive in terms of bandwidth and computational resources. In edge AI applications, determining the minimum amount of events for specific tasks can allow reducing the event rate to improve bandwidth, memory, and processing efficiency. In this paper, we study the effect of event subsampling on the accuracy of event data classification using convolutional neural network (CNN) models. Surprisingly, across various datasets, the number of events per video can be reduced by an order of magnitude with little drop in accuracy, revealing the extent to which we can push the boundaries in accuracy vs. event rate trade-off. Additionally, we also find that lower classification accuracy in high subsampling rates is not solely attributable to information loss due to the subsampling of the events, but that the training of CNNs can be challenging in highly subsampled scenarios, where the sensitivity to hyperparameters increases. We quantify training instability across multiple event-based classification datasets using a novel metric for evaluating the hyperparameter sensitivity of CNNs in different subsampling settings. Finally, we analyze the weight gradients of the network to gain insight into this instability.
Abstract (translated)
活动相机具有低功耗的视觉感知能力，特别适用于边缘设备应用。然而，由于其高事件率，驱动因素是高时间细节，在带宽和计算资源方面可能会有所限制。在边缘人工智能应用中，确定特定任务所需的最低事件量可以降低事件率，提高带宽、内存和处理效率。在本文中，我们研究了事件子采样对使用卷积神经网络（CNN）模型进行事件数据分类的准确性的影响。令人惊讶的是，在各种数据集中，每个视频的事件数量可以降低一个数量级，而准确度几乎没有下降，揭示了我们在准确性和事件率之间的权衡程度。此外，我们还发现，在高度子采样的情况下，较低的分类准确性不仅仅是因为事件子采样导致的，而且是因为在高度子采样情况下训练CNN可能具有挑战性，参数对超参数的敏感性增加。我们使用一种新的指标，评估不同子采样设置中CNN的参数敏感性，来量化训练不稳定。最后，我们分析了网络的权重梯度，以深入了解这种不稳定性。
URL
https://arxiv.org/abs/2409.08953
PDF
-
Scalable Frame Sampling for Video Classification: A Semi-Optimal Policy Approach with Reduced Search Space
2024-09-09 01:11:47Junho Lee, Jeongwoo Shin, Seung Woo Ko, Seongsu Ha, Joonseok LeeAbstract
Given a video with $T$ frames, frame sampling is a task to select $N \ll T$ frames, so as to maximize the performance of a fixed video classifier. Not just brute-force search, but most existing methods suffer from its vast search space of $\binom{T}{N}$, especially when $N$ gets large. To address this challenge, we introduce a novel perspective of reducing the search space from $O(T^N)$ to $O(T)$. Instead of exploring the entire $O(T^N)$ space, our proposed semi-optimal policy selects the top $N$ frames based on the independently estimated value of each frame using per-frame confidence, significantly reducing the computational complexity. We verify that our semi-optimal policy can efficiently approximate the optimal policy, particularly under practical settings. Additionally, through extensive experiments on various datasets and model architectures, we demonstrate that learning our semi-optimal policy ensures stable and high performance regardless of the size of $N$ and $T$.
Abstract (translated)
给定一个具有 $T$ 帧的视频，帧采样是将 $N$ 帧的选择任务，以最大化固定视频分类器的性能。不仅仅是暴力搜索，而且大多数现有方法都存在其巨大的搜索空间 $\binom{T}{N}$，尤其是在 $N$ 较大时。为解决这个问题，我们引入了一种新的观点，将搜索空间从 $O(T^N)$ 减少到 $O(T)$。我们提出的半最优策略基于每个帧的独立估计值，选择前 $N$ 帧，大大减少了计算复杂度。我们验证，我们的半最优策略可以有效地近似最优策略，特别是在实际设置下。此外，通过在各种数据集和模型架构上进行广泛的实验，我们证明，学习我们的半最优策略可以确保稳定和高性能，而不会影响 $N$ 和 $T$ 的值。
URL
https://arxiv.org/abs/2409.05260
PDF
-
ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition
2024-09-03 03:01:25Shiting Xiao, Yuhang Li, Youngeun Kim, Donghyun Lee, Priyadarshini PandaAbstract
Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) for static image tasks such as image classification and segmentation. However, in the more complex video classification domain, SNN-based methods fall considerably short of ANN-based benchmarks due to the challenges in processing dense frame sequences. To bridge this gap, we propose ReSpike, a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost. By decomposing film clips into spatial and temporal components, i.e., RGB image Key Frames and event-like Residual Frames, ReSpike leverages ANN for learning spatial information and SNN for learning temporal information. In addition, we propose a multi-scale cross-attention mechanism for effective feature fusion. Compared to state-of-the-art SNN baselines, our ReSpike hybrid architecture demonstrates significant performance improvements (e.g., >30% absolute accuracy improvement on HMDB-51, UCF-101, and Kinetics-400). Furthermore, ReSpike achieves comparable performance with prior ANN approaches while bringing better accuracy-energy tradeoff.
Abstract (translated)
尖峰神经网络（SNNs）已成为传统人工神经网络（ANNs）在静态图像任务（如图像分类和分割）上的一个引人注目的、能源高效的替代方案。然而，在更复杂的视频分类领域，基于SNN的方法与基于ANN的方法相比，基准测试的结果差距很大，因为处理密集帧序列的挑战。为了弥合这一差距，我们提出了ReSpike，一种结合ANNs和SNNs优势的混合框架，以解决高准确性和低能量成本的动作识别任务。通过将电影剪辑分解为空间和时间组件，即RGB图像关键帧和事件状的残差帧，ReSpike利用ANNs学习空间信息，SNNs学习时间信息。此外，我们还提出了多尺度 cross-attention 机制来实现有效的特征融合。与最先进的SNN基线相比，我们的ReSpike混合架构显示出显著的性能改进（例如，HMDB-51、UCF-101和Kinetics-400的绝对准确度改进率均大于30%）。此外，ReSpike与先前的ANN方法实现了相当的表演，同时带来了更好的准确-能量权衡。
URL
https://arxiv.org/abs/2409.01564
PDF
-
Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification
2024-08-26 17:33:47Mahrukh Awan, Asmar Nadeem, Muhammad Junaid Awan, Armin Mustafa, Syed Sameed HusainAbstract
Exploiting both audio and visual modalities for video classification is a challenging task, as the existing methods require large model architectures, leading to high computational complexity and resource requirements. Smaller architectures, on the other hand, struggle to achieve optimal performance. In this paper, we propose Attend-Fusion, an audio-visual (AV) fusion approach that introduces a compact model architecture specifically designed to capture intricate audio-visual relationships in video data. Through extensive experiments on the challenging YouTube-8M dataset, we demonstrate that Attend-Fusion achieves an F1 score of 75.64\% with only 72M parameters, which is comparable to the performance of larger baseline models such as Fully-Connected Late Fusion (75.96\% F1 score, 341M parameters). Attend-Fusion achieves similar performance to the larger baseline model while reducing the model size by nearly 80\%, highlighting its efficiency in terms of model complexity. Our work demonstrates that the Attend-Fusion model effectively combines audio and visual information for video classification, achieving competitive performance with significantly reduced model size. This approach opens new possibilities for deploying high-performance video understanding systems in resource-constrained environments across various applications.
Abstract (translated)
利用音频和视频模态进行视频分类是一个具有挑战性的任务，因为现有的方法需要大型模型架构，导致高计算复杂度和资源需求。较小的架构则很难实现最优性能。在本文中，我们提出了Attend-Fusion，一种专为捕捉视频数据中复杂的音频-视觉关系而设计的音频-视觉（AV）融合方法。通过对具有挑战性的YouTube-8M数据集的广泛实验，我们证明了Attend-Fusion达到75.64\%的F1得分，仅使用72M参数，与具有类似性能的大型基线模型（如Fully-Connected Late Fusion，75.96\% F1 score，341M parameters）相当。Attend-Fusion在大型基线模型的同时减小了模型大小，凸显了其在模型复杂度方面的效率。我们的工作表明，Attend-Fusion模型能够有效地结合音频和视频信息进行视频分类，实现与显著减小模型大小相当的竞争性能。这种方法为在各种应用环境中部署高效的视频理解系统提供了新的可能性。
URL
https://arxiv.org/abs/2408.14441
PDF
-
VidModEx: Interpretable and Efficient Black Box Model Extraction for High-Dimensional Spaces
2024-08-04 20:38:45Somnath Sendhil Kumar, Yuvaraj Govindarajulu, Pavan Kulkarni, Manojkumar ParmarAbstract
In the domain of black-box model extraction, conventional methods reliant on soft labels or surrogate datasets struggle with scaling to high-dimensional input spaces and managing the complexity of an extensive array of interrelated classes. In this work, we present a novel approach that utilizes SHAP (SHapley Additive exPlanations) to enhance synthetic data generation. SHAP quantifies the individual contributions of each input feature towards the victim model's output, facilitating the optimization of an energy-based GAN towards a desirable output. This method significantly boosts performance, achieving a 16.45% increase in the accuracy of image classification models and extending to video classification models with an average improvement of 26.11% and a maximum of 33.36% on challenging datasets such as UCF11, UCF101, Kinetics 400, Kinetics 600, and Something-Something V2. We further demonstrate the effectiveness and practical utility of our method under various scenarios, including the availability of top-k prediction probabilities, top-k prediction labels, and top-1 labels.
Abstract (translated)
在领域黑盒模型提取中，传统方法在软标签或代理数据集上依赖，很难扩展到高维输入空间并管理广泛的相互类别的复杂性。在本文中，我们提出了一个新方法，利用SHAP（SHapley Additive exPlanations）增强合成数据生成。SHAP计算每个输入特征对受害者模型输出的单独贡献，从而促进基于能量的GAN优化到一个理想的输出。这种方法显著提高了性能，将图像分类模型的准确率提高了16.45%，并将视频分类模型的平均改进扩展至具有挑战性的数据集（如UCF11，UCF101，Kinetics 400，Kinetics 600和Something-Something V2），这些数据集的准确率提高至33.36%。我们进一步展示了我们的方法在不同场景下的有效性和实际应用价值，包括存在Top-k预测概率、Top-k预测标签和Top-1标签的情况。
URL
https://arxiv.org/abs/2408.02140
PDF
-
Learning Video Context as Interleaved Multimodal Sequences
2024-07-31 17:23:57Kevin Qinghong Lin, Pengchuan Zhang, Difei Gao, Xide Xia, Joya Chen, Ziteng Gao, Jinheng Xie, Xuhong Xiao, Mike Zheng ShouAbstract
Narrative videos, such as movies, pose significant challenges in video understanding due to their rich contexts (characters, dialogues, storylines) and diverse demands (identify who, relationship, and reason). In this paper, we introduce MovieSeq, a multimodal language model developed to address the wide range of challenges in understanding video contexts. Our core idea is to represent videos as interleaved multimodal sequences (including images, plots, videos, and subtitles), either by linking external knowledge databases or using offline models (such as whisper for subtitles). Through instruction-tuning, this approach empowers the language model to interact with videos using interleaved multimodal instructions. For example, instead of solely relying on video as input, we jointly provide character photos alongside their names and dialogues, allowing the model to associate these elements and generate more comprehensive responses. To demonstrate its effectiveness, we validate MovieSeq's performance on six datasets (LVU, MAD, Movienet, CMD, TVC, MovieQA) across five settings (video classification, audio description, video-text retrieval, video captioning, and video question-answering). The code will be public at this https URL.
Abstract (translated)
叙事视频（如电影）在视频理解方面面临着显著的挑战，因为它们具有丰富的上下文（角色、对话、故事情节）和多样化的要求（确定谁、关系和原因）。在本文中，我们介绍了MovieSeq，一种多模态语言模型，旨在解决理解视频上下文的广泛范围挑战。我们的核心思想是将视频表示为交织的多模态序列（包括图像、剧情、视频和字幕），或者通过链接外部知识数据库或使用离线模型（如whisper for subtitles）来实现。通过指令调整，这种方法使语言模型能够使用交织的多模态指令与视频进行交互。例如， instead of solely relying on video as input, we jointly provide character photos alongside their names and dialogues, allowing the model to associate these elements and generate more comprehensive responses. 为了证明其有效性，我们在六个数据集（LVU，MAD，Movienet，CMD，TVC，MovieQA）和五个设置（视频分类、音频描述、视频-文本检索、视频字幕、视频问题回答）上对MovieSeq进行了评估。代码将在这个https URL上公开。
URL
https://arxiv.org/abs/2407.21757
PDF
-
MultiHateClip: A Multilingual Benchmark Dataset for Hateful Video Detection on YouTube and Bilibili
2024-07-28 08:19:09Han Wang, Tan Rui Yang, Usman Naseem, Roy Ka-Wei LeeAbstract
Hate speech is a pressing issue in modern society, with significant effects both online and offline. Recent research in hate speech detection has primarily centered on text-based media, largely overlooking multimodal content such as videos. Existing studies on hateful video datasets have predominantly focused on English content within a Western context and have been limited to binary labels (hateful or non-hateful), lacking detailed contextual information. This study presents MultiHateClip1 , an novel multilingual dataset created through hate lexicons and human annotation. It aims to enhance the detection of hateful videos on platforms such as YouTube and Bilibili, including content in both English and Chinese languages. Comprising 2,000 videos annotated for hatefulness, offensiveness, and normalcy, this dataset provides a cross-cultural perspective on gender-based hate speech. Through a detailed examination of human annotation results, we discuss the differences between Chinese and English hateful videos and underscore the importance of different modalities in hateful and offensive video analysis. Evaluations of state-of-the-art video classification models, such as VLM, GPT-4V and Qwen-VL, on MultiHateClip highlight the existing challenges in accurately distinguishing between hateful and offensive content and the urgent need for models that are both multimodally and culturally nuanced. MultiHateClip represents a foundational advance in enhancing hateful video detection by underscoring the necessity of a multimodal and culturally sensitive approach in combating online hate speech.
Abstract (translated)
仇恨言论是现代社会的一个紧迫问题，对在线和离线领域都产生了重大影响。最近，仇恨言论检测的研究主要集中在基于文本的媒体上，很大程度上忽略了多模态内容，如视频。现有的仇恨视频数据集研究主要集中在西方语境下的英语内容，并局限于二元标签（仇恨或非仇恨），缺乏详细的上下文信息。本研究提出了MultiHateClip1，一个由仇恨词汇和人类标注创建的多语言数据集。它旨在提高YouTube和Bilibili等平台上的仇恨视频检测，包括英语和中文内容。这个数据集包括2000个被标注为仇恨性、攻击性和正常性的视频，为基于性别仇恨言论提供了一个跨文化视角。通过详细审查人类标注结果，我们讨论了中文和英文仇恨视频之间的差异，并强调了两类视频分析中不同模态的重要性。评估了最先进的视频分类模型，如VLM、GPT-4V和Qwen-VL，在MultiHateClip上的效果，突出了准确区分仇恨和攻击性内容以及迫切需要具有多模态和文化细微的模型。MultiHateClip在增强仇恨视频检测方面取得了基础性的进展，强调在打击网络仇恨言论方面需要多模态和敏感的方法。
URL
https://arxiv.org/abs/2408.03468
PDF
-
UniForensics: Face Forgery Detection via General Facial Representation
2024-07-26 20:51:54Ziyuan Fang, Hanqing Zhao, Tianyi Wei, Wenbo Zhou, Ming Wan, Zhanyi Wang, Weiming Zhang, Nenghai YuAbstract
Previous deepfake detection methods mostly depend on low-level textural features vulnerable to perturbations and fall short of detecting unseen forgery methods. In contrast, high-level semantic features are less susceptible to perturbations and not limited to forgery-specific artifacts, thus having stronger generalization. Motivated by this, we propose a detection method that utilizes high-level semantic features of faces to identify inconsistencies in temporal domain. We introduce UniForensics, a novel deepfake detection framework that leverages a transformer-based video classification network, initialized with a meta-functional face encoder for enriched facial representation. In this way, we can take advantage of both the powerful spatio-temporal model and the high-level semantic information of faces. Furthermore, to leverage easily accessible real face data and guide the model in focusing on spatio-temporal features, we design a Dynamic Video Self-Blending (DVSB) method to efficiently generate training samples with diverse spatio-temporal forgery traces using real facial videos. Based on this, we advance our framework with a two-stage training approach: The first stage employs a novel self-supervised contrastive learning, where we encourage the network to focus on forgery traces by impelling videos generated by the same forgery process to have similar representations. On the basis of the representation learned in the first stage, the second stage involves fine-tuning on face forgery detection dataset to build a deepfake detector. Extensive experiments validates that UniForensics outperforms existing face forgery methods in generalization ability and robustness. In particular, our method achieves 95.3\% and 77.2\% cross dataset AUC on the challenging Celeb-DFv2 and DFDC respectively.
Abstract (translated)
先前的深度伪造检测方法主要依赖于低水平的纹理特征，这些特征易受扰动，并且无法检测出未被标识为伪造的方法。相反，高层面的语义特征对扰动更敏感，不受伪造特定瑕疵的限制，因此具有更强的一般化能力。基于这一点，我们提出了利用人脸面部高层语义特征来识别时域中存在差异的检测方法。我们引入了UniForensics，这是一种新颖的深度伪造检测框架，利用Transformer类视频分类网络作为启动器，该网络初始化了一个元功能脸部编码器，以丰富面部表示。通过这种方式，我们可以充分利用强大的时空模型和面部高级语义信息的优势。此外，为了利用易于访问的真实人脸数据，并引导模型集中注意力于时空特征，我们设计了一个动态视频自融合（DVSB）方法，有效地使用真实的人脸视频生成多样化的时空伪造痕迹训练样本，从而在第二阶段引入了深度学习框架的改进：首先，我们提出了一个新颖的监督对抗性学习策略，通过促使从同一伪造过程中产生的视频具有相似表示，来引导网络关注伪造足迹。基于第一阶段所学到的代表，在第二阶段中涉及对人脸伪造检测数据集进行微调以构建深度伪造检测器。广泛的实验验证了UniForensics在一般化能力和鲁棒性方面优于现有的面部伪造方法。具体来说，我们的方法在挑战性的Celeb-DFv2和DFDC数据集中分别实现95.3\%和77.2\%的交叉数据AUC。 总结：利用高层面的语义特征检测深度伪造是一个潜在的研究领域，并且可以增强现有的框架能力。我们提出了一个综合解决方案，结合了自监督强化学习策略与分层训练方法，以改进深度伪造检测器的表现。
URL
https://arxiv.org/abs/2407.19079
PDF
-
LookupViT: Compressing visual information to a limited number of tokens
2024-07-17 17:22:43Rajat Koner, Gagan Jain, Prateek Jain, Volker Tresp, Sujoy PaulAbstract
Vision Transformers (ViT) have emerged as the de-facto choice for numerous industry grade vision solutions. But their inference cost can be prohibitive for many settings, as they compute self-attention in each layer which suffers from quadratic computational complexity in the number of tokens. On the other hand, spatial information in images and spatio-temporal information in videos is usually sparse and redundant. In this work, we introduce LookupViT, that aims to exploit this information sparsity to reduce ViT inference cost. LookupViT provides a novel general purpose vision transformer block that operates by compressing information from higher resolution tokens to a fixed number of tokens. These few compressed tokens undergo meticulous processing, while the higher-resolution tokens are passed through computationally cheaper layers. Information sharing between these two token sets is enabled through a bidirectional cross-attention mechanism. The approach offers multiple advantages - (a) easy to implement on standard ML accelerators (GPUs/TPUs) via standard high-level operators, (b) applicable to standard ViT and its variants, thus generalizes to various tasks, (c) can handle different tokenization and attention approaches. LookupViT also offers flexibility for the compressed tokens, enabling performance-computation trade-offs in a single trained model. We show LookupViT's effectiveness on multiple domains - (a) for image-classification (ImageNet-1K and ImageNet-21K), (b) video classification (Kinetics400 and Something-Something V2), (c) image captioning (COCO-Captions) with a frozen encoder. LookupViT provides $2\times$ reduction in FLOPs while upholding or improving accuracy across these domains. In addition, LookupViT also demonstrates out-of-the-box robustness and generalization on image classification (ImageNet-C,R,A,O), improving by up to $4\%$ over ViT.
Abstract (translated)
Vision Transformers (ViT) 已经成为许多行业级视觉解决方案的首选，但对于许多设置，其推理成本可能过高，因为它们在每层计算自注意力，这会导致在token数量上的平方计算复杂度。另一方面，图像和视频中的空间信息通常是稀疏和冗余的。在这项工作中，我们介绍了一个名为LookupViT的工作，旨在利用这种信息稀疏性降低ViT的推理成本。LookupViT提供了一个新颖的一般性视觉 transformer 块，通过压缩高分辨率token的信息，将其压缩到固定数量的路上。这几个经过精心处理的压缩token在较低计算成本的层中通过双向注意力机制进行信息共享。这种方法具有多个优点-（a）通过标准的高层操作很容易实现于标准 ML 加速器（GPUs/TPUs）上，（b）适用于标准的ViT及其变体，所以可以应用于各种任务，（c）可以处理不同的token化和注意力方法。LookupViT还具有压缩token的灵活性，在单个训练模型上实现性能计算权衡。我们在多个领域证明了LookupViT的有效性 -（a）图像分类（ImageNet-1K 和 ImageNet-21K），（b）视频分类（Kinetics400 和 Something-Something V2），（c）图像摘要（COCO-Captions）使用冻结编码器。LookupViT在FLOPs方面提供了2倍的增长，同时在这些领域保持或提高准确性。此外，LookupViT还展示了在图像分类（ImageNet-C，R，A，O）上的稳健性和泛化能力，提高了4%以上。
URL
https://arxiv.org/abs/2407.12753
PDF
-
Learning Natural Consistency Representation for Face Forgery Video Detection
2024-07-15 09:00:02Daichi Zhang, Zihao Xiao, Shikun Li, Fanzhao Lin, Jianmin Li, Shiming GeAbstract
Face Forgery videos have elicited critical social public concerns and various detectors have been proposed. However, fully-supervised detectors may lead to easily overfitting to specific forgery methods or videos, and existing self-supervised detectors are strict on auxiliary tasks, such as requiring audio or multi-modalities, leading to limited generalization and robustness. In this paper, we examine whether we can address this issue by leveraging visual-only real face videos. To this end, we propose to learn the Natural Consistency representation (NACO) of real face videos in a self-supervised manner, which is inspired by the observation that fake videos struggle to maintain the natural spatiotemporal consistency even under unknown forgery methods and different perturbations. Our NACO first extracts spatial features of each frame by CNNs then integrates them into Transformer to learn the long-range spatiotemporal representation, leveraging the advantages of CNNs and Transformer on local spatial receptive field and long-term memory respectively. Furthermore, a Spatial Predictive Module~(SPM) and a Temporal Contrastive Module~(TCM) are introduced to enhance the natural consistency representation learning. The SPM aims to predict random masked spatial features from spatiotemporal representation, and the TCM regularizes the latent distance of spatiotemporal representation by shuffling the natural order to disturb the consistency, which could both force our NACO more sensitive to the natural spatiotemporal consistency. After the representation learning stage, a MLP head is fine-tuned to perform the usual forgery video classification task. Extensive experiments show that our method outperforms other state-of-the-art competitors with impressive generalization and robustness.
Abstract (translated)
面对伪造视频，已经引起了广泛的社会公众关注，并提出了各种检测方法。然而，完全监督的检测可能会导致对特定伪造方法的过拟合，而现有的自监督检测方法对辅助任务有严格的要求，导致对泛化和鲁棒性的限制。在本文中，我们探讨是否可以通过利用仅包含视觉信息的真实人脸视频来解决这个问题。为此，我们提出了一种自监督地学习真实人脸视频的自然一致性表示（NACO）的方法，这是基于观察到假视频在未知伪造方法下甚至无法保持自然时空一致性的启示。我们的NACO首先通过CNN提取每个帧的时空特征，然后将它们整合到Transformer中学习长距离时空表示，利用CNN和Transformer在局部时空感受野和长期记忆方面的优势。此外，还引入了空间预测模块（SPM）和时间对比模块（TCM）来增强自然一致性表示的学习。SPM旨在预测从时空表示中随机遮罩的时空特征，而TCM通过随机打乱自然顺序来干扰一致性，这可能会使我们的人脸NACO更加敏感。在表示学习阶段之后，对MLP头进行微调以执行通常的伪造视频分类任务。大量实验证明，我们的方法在泛化能力和鲁棒性方面优于其他最先进的竞争对手。
URL
https://arxiv.org/abs/2407.10550
PDF
