Video_Classification
-
Open Vocabulary Multi-Label Video Classification
2024-07-12 07:53:54Rohit Gupta, Mamshad Nayeem Rizve, Jayakrishnan Unnikrishnan, Ashish Tawari, Son Tran, Mubarak Shah, Benjamin Yao, Trishul ChilimbiAbstract
Pre-trained vision-language models (VLMs) have enabled significant progress in open vocabulary computer vision tasks such as image classification, object detection and image segmentation. Some recent works have focused on extending VLMs to open vocabulary single label action classification in videos. However, previous methods fall short in holistic video understanding which requires the ability to simultaneously recognize multiple actions and entities e.g., objects in the video in an open vocabulary setting. We formulate this problem as open vocabulary multilabel video classification and propose a method to adapt a pre-trained VLM such as CLIP to solve this task. We leverage large language models (LLMs) to provide semantic guidance to the VLM about class labels to improve its open vocabulary performance with two key contributions. First, we propose an end-to-end trainable architecture that learns to prompt an LLM to generate soft attributes for the CLIP text-encoder to enable it to recognize novel classes. Second, we integrate a temporal modeling module into CLIP's vision encoder to effectively model the spatio-temporal dynamics of video concepts as well as propose a novel regularized finetuning technique to ensure strong open vocabulary classification performance in the video domain. Our extensive experimentation showcases the efficacy of our approach on multiple benchmark datasets.
Abstract (translated)
预训练的视觉-语言模型（VLMs）已经在诸如图像分类、目标检测和图像分割等开放词汇计算机视觉任务中取得了显著的进展。一些最近的工作将重点放在将VLMs扩展到开放词汇单标签动作分类在视频上。然而，之前的方法在全面理解视频方面存在缺陷，这需要同时识别多个动作和实体，例如视频中的对象。我们将这个问题转化为开放词汇多标签视频分类，并提出了将预训练的VLM如CLIP适应这种任务的方法。我们利用大型语言模型（LLMs）为VLM提供语义指导，以便提高其开放词汇性能。我们的方法具有两个关键贡献：第一，我们提出了一个端到端的训练架构，学习提示LLM生成软特征，以使它能够识别新的类别。第二，我们将一个时间建模模块集成到CLIP的视觉编码器中，有效地建模了视频概念的空间和时间动态，并提出了一个新的正则化微调技术，以确保在视频域中具有强大的开放词汇分类性能。我们对多个基准数据集的实验表明，我们的方法在多个方面都具有有效性。
URL
https://arxiv.org/abs/2407.09073
PDF
-
PUDD: Towards Robust Multi-modal Prototype-based Deepfake Detection
2024-06-22 19:21:42Alvaro Lopez Pellcier, Yi Li, Plamen AngelovAbstract
Deepfake techniques generate highly realistic data, making it challenging for humans to discern between actual and artificially generated images. Recent advancements in deep learning-based deepfake detection methods, particularly with diffusion models, have shown remarkable progress. However, there is a growing demand for real-world applications to detect unseen individuals, deepfake techniques, and scenarios. To address this limitation, we propose a Prototype-based Unified Framework for Deepfake Detection (PUDD). PUDD offers a detection system based on similarity, comparing input data against known prototypes for video classification and identifying potential deepfakes or previously unseen classes by analyzing drops in similarity. Our extensive experiments reveal three key findings: (1) PUDD achieves an accuracy of 95.1% on Celeb-DF, outperforming state-of-the-art deepfake detection methods; (2) PUDD leverages image classification as the upstream task during training, demonstrating promising performance in both image classification and deepfake detection tasks during inference; (3) PUDD requires only 2.7 seconds for retraining on new data and emits 10$^{5}$ times less carbon compared to the state-of-the-art model, making it significantly more environmentally friendly.
Abstract (translated)
深度伪造技术生成的数据非常逼真，使得人类很难区分真实和伪造的图像。在深度学习为基础的深度伪造检测方法的最新进展中，特别是扩散模型，取得了显著的进步。然而，对于检测未见过的个体、深度伪造技术和情景，现实世界的应用需求不断增加。为了克服这一局限，我们提出了一个基于原型统一的深度伪造检测框架（PUDD）。PUDD 提供了一种基于相似度的检测系统，将输入数据与已知原型进行比较，通过分析相似性的下降来识别潜在的深度伪造者或以前未见过的类别。我们进行的广泛实验揭示了三个关键发现：（1）PUDD 在Celeb-DF上的准确率达到了95.1%，超越了最先进的深度伪造检测方法；（2）在训练过程中将图像分类作为上游任务，表明在推理过程中在图像分类和深度伪造检测任务上都取得了有前景的表现；（3）在只使用新的数据进行重新训练的情况下，PUDD 只需要2.7秒钟，排放的碳比最先进模型少10^5倍，使得它具有显著的环保优势。
URL
https://arxiv.org/abs/2406.15921
PDF
-
MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearning
2024-06-21 00:13:17Jiali Cheng, Hadi AmiriAbstract
Recent advancements in Machine Unlearning (MU) have introduced solutions to selectively remove certain training samples, such as those with outdated or sensitive information, from trained models. Despite these advancements, evaluation of MU methods have been inconsistent, employing different trained models and architectures, and sample removal strategies, which hampers accurate comparison. In addition, prior MU approaches have mainly focused on singular tasks or modalities, which is not comprehensive. To address these limitations, we develop MU-Bench, the first comprehensive benchmark for MU that (i) unifies the sets of deleted samples and trained models, and (ii) provides broad coverage of tasks and data modalities, including previously unexplored domains such as speech and video classification. Our evaluation show that RandLabel and SalUn are the most effective general MU approaches on MU-Bench, and BadT and SCRUB are capable of achieving random performance on the deletion set. We analyze several under-investigated aspects of unlearning, including scalability, the impacts of parameter-efficient fine-tuning and curriculum learning, and susceptibility to dataset biases. MU-Bench provides an easy-to-use package that includes dataset splits, models, and implementations, together with a leader board to enable unified and scalable MU research.
Abstract (translated)
近年来，机器学习（ML）中的进展为选择性地从训练模型中移除某些训练样本，如过时或敏感信息，提供了解决方案。尽管这些进步，但ML方法的评估存在不一致性，使用了不同的训练模型和架构以及样本删除策略，这阻碍了准确比较。此外，之前的多元方法主要集中在单一任务或维度上，不全面。为了克服这些限制，我们开发了MU-Bench，第一个全面的MU基准，它统一了删除的样本和训练模型，并为任务和数据维度提供广泛的覆盖，包括之前未探索的领域，如语音和视频分类。我们的评估显示，RandLabel和SalUn是MU-Bench中最有效的通用ML方法，而BadT和SCRUB在删除集上具有随机性能。我们分析了几个尚未进行深入研究的问题，包括可扩展性、参数高效微调以及正则化学习，以及易受数据偏差的影响。MU-Bench提供了一个易于使用的软件包，包括数据集划分、模型和实现，以及领导者板，以实现统一和可扩展的ML研究。
URL
https://arxiv.org/abs/2406.14796
PDF
-
DL-KDD: Dual-Light Knowledge Distillation for Action Recognition in the Dark
2024-06-04 16:38:06Chi-Jui Chang, Oscar Tai-Yuan Chen, Vincent S. TsengAbstract
Human action recognition in dark videos is a challenging task for computer vision. Recent research focuses on applying dark enhancement methods to improve the visibility of the video. However, such video processing results in the loss of critical information in the original (un-enhanced) video. Conversely, traditional two-stream methods are capable of learning information from both original and processed videos, but it can lead to a significant increase in the computational cost during the inference phase in the task of video classification. To address these challenges, we propose a novel teacher-student video classification framework, named Dual-Light KnowleDge Distillation for Action Recognition in the Dark (DL-KDD). This framework enables the model to learn from both original and enhanced video without introducing additional computational cost during inference. Specifically, DL-KDD utilizes the strategy of knowledge distillation during training. The teacher model is trained with enhanced video, and the student model is trained with both the original video and the soft target generated by the teacher model. This teacher-student framework allows the student model to predict action using only the original input video during inference. In our experiments, the proposed DL-KDD framework outperforms state-of-the-art methods on the ARID, ARID V1.5, and Dark-48 datasets. We achieve the best performance on each dataset and up to a 4.18% improvement on Dark-48, using only original video inputs, thus avoiding the use of two-stream framework or enhancement modules for inference. We further validate the effectiveness of the distillation strategy in ablative experiments. The results highlight the advantages of our knowledge distillation framework in dark human action recognition.
Abstract (translated)
人类动作识别在黑暗视频中是一项具有挑战性的计算机视觉任务。最近的研究专注于将黑暗增强方法应用于提高视频的可视化度。然而，这种视频处理会导致原始（未增强）视频中关键信息的丢失。相反，传统的两流方法可以从原始和处理视频中学到信息，但在推理阶段会显著增加计算成本。为了应对这些挑战，我们提出了一个名为Dual-Light KnowleDge Distillation for Action Recognition in the Dark (DL-KDD)的新教师-学生视频分类框架。这个框架在推理过程中不引入额外的计算成本。具体来说，DL-KDD利用知识蒸馏策略进行训练。教师模型通过增强视频进行训练，学生模型通过原始视频和教师模型生成的软目标进行训练。这个教师-学生框架使得学生模型在推理过程中仅使用原始输入视频预测动作。在实验中，我们发现，与最先进的Methods相比，DL-KDD框架在ARID、ARID V1.5和Dark-48数据集上的性能都表现出色。我们在每个数据集上都实现了最佳性能，而在Dark-48数据集上实现了4.18%的提高，而使用 only the original video inputs。我们还进一步验证了知识蒸馏策略在实验中的有效性。结果强调了我们在黑暗中人类动作识别中使用知识蒸馏框架的优势。
URL
https://arxiv.org/abs/2406.02468
PDF
-
Few-Shot Classification of Interactive Activities of Daily Living
2024-06-03 17:59:55Zane Durante, Robathan Harries, Edward Vendrow, Zelun Luo, Yuta Kyuragi, Kazuki Kozuka, Li Fei-Fei, Ehsan AdeliAbstract
Understanding Activities of Daily Living (ADLs) is a crucial step for different applications including assistive robots, smart homes, and healthcare. However, to date, few benchmarks and methods have focused on complex ADLs, especially those involving multi-person interactions in home environments. In this paper, we propose a new dataset and benchmark, InteractADL, for understanding complex ADLs that involve interaction between humans (and objects). Furthermore, complex ADLs occurring in home environments comprise a challenging long-tailed distribution due to the rarity of multi-person interactions, and pose fine-grained visual recognition tasks due to the presence of semantically and visually similar classes. To address these issues, we propose a novel method for fine-grained few-shot video classification called Name Tuning that enables greater semantic separability by learning optimal class name vectors. We show that Name Tuning can be combined with existing prompt tuning strategies to learn the entire input text (rather than only learning the prompt or class names) and demonstrate improved performance for few-shot classification on InteractADL and 4 other fine-grained visual classification benchmarks. For transparency and reproducibility, we release our code at this https URL.
Abstract (translated)
理解日常活动活动（ADLs）对于诸如辅助机器人、智能家居和医疗设备等各个应用领域来说至关重要。然而，到目前为止，很少有基准和方法关注复杂ADL，尤其是涉及多人在家庭环境中的交互。在本文中，我们提出了一个新的数据集和基准，称为交互ADL，以了解复杂ADL中的人（及其与物体）之间的交互。此外，家庭环境中的复杂ADL由于多个人类交互的稀有性，构成了一个具有挑战性的长尾分布，并提出了细粒度的视觉识别任务。为了应对这些问题，我们提出了一个名为Name Tuning的新方法，一种通过学习最优分类名称向量来提高语义分离的细粒度视觉分类方法。我们证明了Name Tuning可以与现有的提示调整策略相结合，以学习整个输入文本（而不仅仅是提示或分类名称），并在InteractADL和四个其他细粒度视觉分类基准上取得了较好的性能。为了保证透明度和可重复性，我们将代码发布在此处：https://www.example.com/。
URL
https://arxiv.org/abs/2406.01662
PDF
-
ToxVidLLM: A Multimodal LLM-based Framework for Toxicity Detection in Code-Mixed Videos
2024-05-31 05:40:56Krishanu Maity, A. S. Poornash, Sriparna Saha, Pushpak BhattacharyyaAbstract
In an era of rapidly evolving internet technology, the surge in multimodal content, including videos, has expanded the horizons of online communication. However, the detection of toxic content in this diverse landscape, particularly in low-resource code-mixed languages, remains a critical challenge. While substantial research has addressed toxic content detection in textual data, the realm of video content, especially in non-English languages, has been relatively underexplored. This paper addresses this research gap by introducing a benchmark dataset, the first of its kind, consisting of 931 videos with 4021 code-mixed Hindi-English utterances collected from YouTube. Each utterance within this dataset has been meticulously annotated for toxicity, severity, and sentiment labels. We have developed an advanced Multimodal Multitask framework built for Toxicity detection in Video Content by leveraging Large Language Models (LLMs), crafted for the primary objective along with the additional tasks of conducting sentiment and severity analysis. ToxVidLLM incorporates three key modules the Encoder module, Cross-Modal Synchronization module, and Multitask module crafting a generic multimodal LLM customized for intricate video classification tasks. Our experiments reveal that incorporating multiple modalities from the videos substantially enhances the performance of toxic content detection by achieving an Accuracy and Weighted F1 score of 94.29% and 94.35%, respectively.
Abstract (translated)
在快速发展的互联网技术时代，多模态内容的激增已经扩展了互联网沟通的视野。然而，在多样性较高的场景中检测有毒内容，特别是低资源代码混合语言中的内容，仍然是一个关键的挑战。尽管在文本数据中的有毒内容检测方面已经进行了大量研究，但在视频内容领域，尤其是在非英语语言中，这个领域仍然相对缺乏研究。本文通过引入一个基准数据集来填补这个研究空白，这个数据集包括从YouTube上收集的931个视频，每个视频都有4021个印度-英语混合代码的语料。这个数据集中的每个语料都已详细注释为毒性、严重性和情感标签。我们利用大型语言模型（LLMs）构建了用于视频内容中有毒内容检测的先进多模态多任务框架，并专门为情感和严重性分析增加了任务。ToxVidLLM包括三个关键模块：编码器模块、跨模态同步模块和多任务模块，为复杂的视频分类任务创建了一个通用的多模态LLM。我们的实验结果表明，通过引入视频中的多个模态，可以极大地提高有毒内容检测的性能，达到94.29%的准确度和94.35%的加权F1分数。
URL
https://arxiv.org/abs/2405.20628
PDF
-
DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmark
2024-05-30 05:36:12Haoxing Chen, Yan Hong, Zizheng Huang, Zhuoer Xu, Zhangxuan Gu, Yaohui Li, Jun Lan, Huijia Zhu, Jianfu Zhang, Weiqiang Wang, Huaxiong LiAbstract
Recently, video generation techniques have advanced rapidly. Given the popularity of video content on social media platforms, these models intensify concerns about the spread of fake information. Therefore, there is a growing demand for detectors capable of distinguishing between fake AI-generated videos and mitigating the potential harm caused by fake information. However, the lack of large-scale datasets from the most advanced video generators poses a barrier to the development of such detectors. To address this gap, we introduce the first AI-generated video detection dataset, GenVideo. It features the following characteristics: (1) a large volume of videos, including over one million AI-generated and real videos collected; (2) a rich diversity of generated content and methodologies, covering a broad spectrum of video categories and generation techniques. We conducted extensive studies of the dataset and proposed two evaluation methods tailored for real-world-like scenarios to assess the detectors' performance: the cross-generator video classification task assesses the generalizability of trained detectors on generators; the degraded video classification task evaluates the robustness of detectors to handle videos that have degraded in quality during dissemination. Moreover, we introduced a plug-and-play module, named Detail Mamba (DeMamba), designed to enhance the detectors by identifying AI-generated videos through the analysis of inconsistencies in temporal and spatial dimensions. Our extensive experiments demonstrate DeMamba's superior generalizability and robustness on GenVideo compared to existing detectors. We believe that the GenVideo dataset and the DeMamba module will significantly advance the field of AI-generated video detection. Our code and dataset will be aviliable at \url{this https URL}.
Abstract (translated)
近年来，视频生成技术发展迅速。考虑到社交媒体上视频内容的流行，这些模型加剧了对虚假信息传播的担忧。因此，人们越来越需要能够区分虚假AI生成的视频并减轻虚假信息可能带来的潜在危害的检测器。然而，从最先进的视频生成器那里获取的大规模数据集的缺乏构成了发展这种检测器的障碍。为解决这个空白，我们引入了第一个AI生成的视频检测数据集GenVideo。其特征如下：（1）大量的视频，包括超过100万AI生成和真实视频；（2）丰富多样的生成内容和方法论，涵盖了广泛的视频类别和生成技术。我们对该数据集进行了广泛的研究，并提出了两个针对现实场景的评估方法，以评估检测器的性能：跨生成器视频分类任务评估训练后的检测器的泛化能力；降低视频分类任务评估检测器对质量下降的视频的鲁棒性。此外，我们还引入了一个名为Detail Mamba（DeMamba）的插件模块，通过分析时空维度的不一致性来识别AI生成的视频。我们的大量实验证明，DeMamba在GenVideo数据集上比现有检测器具有更卓越的泛化能力和鲁棒性。我们相信，GenVideo数据集和DeMamba模块将对AI生成视频检测领域产生重大影响。我们的代码和数据集将在此处可用：https://this URL。
URL
https://arxiv.org/abs/2405.19707
PDF
-
SIAVC: Semi-Supervised Framework for Industrial Accident Video Classification
2024-05-23 12:44:51Zuoyong Li, Qinghua Lin, Haoyi Fan, Tiesong Zhao, David ZhangAbstract
Semi-supervised learning suffers from the imbalance of labeled and unlabeled training data in the video surveillance scenario. In this paper, we propose a new semi-supervised learning method called SIAVC for industrial accident video classification. Specifically, we design a video augmentation module called the Super Augmentation Block (SAB). SAB adds Gaussian noise and randomly masks video frames according to historical loss on the unlabeled data for model optimization. Then, we propose a Video Cross-set Augmentation Module (VCAM) to generate diverse pseudo-label samples from the high-confidence unlabeled samples, which alleviates the mismatch of sampling experience and provides high-quality training data. Additionally, we construct a new industrial accident surveillance video dataset with frame-level annotation, namely ECA9, to evaluate our proposed method. Compared with the state-of-the-art semi-supervised learning based methods, SIAVC demonstrates outstanding video classification performance, achieving 88.76\% and 89.13\% accuracy on ECA9 and Fire Detection datasets, respectively. The source code and the constructed dataset ECA9 will be released in \url{this https URL}.
Abstract (translated)
半监督学习在视频监控场景中受到有标签和无标签训练数据之间不平衡的影响。在本文中，我们提出了一种名为SIAVC的新半监督学习方法，用于工业事故视频分类。具体来说，我们设计了一个视频增强模块，称为Super Augmentation Block（SAB）。SAB在无标签数据上根据历史损失添加高斯噪声并随机遮盖视频帧。然后，我们提出了一个视频跨样本增强模块（VCAM），从高置信度的无标签样本中生成多样性的伪标签样本，减轻了抽样体验的差异，并为高质量训练数据提供了支持。此外，我们还构建了一个新的工业事故监控视频数据集，以评估我们所提出的方法。与最先进的半监督学习方法相比，SIAVC表现出出色的视频分类性能，在ECA9和火灾检测数据集上分别获得了88.76%和89.13%的准确率。源代码和构建的数据集ECA9将发布在https://这个URL上。
URL
https://arxiv.org/abs/2405.14506
PDF
-
Towards Gradient-based Time-Series Explanations through a SpatioTemporal Attention Network
2024-05-18 02:36:29Min Hun LeeAbstract
In this paper, we explore the feasibility of using a transformer-based, spatiotemporal attention network (STAN) for gradient-based time-series explanations. First, we trained the STAN model for video classifications using the global and local views of data and weakly supervised labels on time-series data (i.e. the type of an activity). We then leveraged a gradient-based XAI technique (e.g. saliency map) to identify salient frames of time-series data. According to the experiments using the datasets of four medically relevant activities, the STAN model demonstrated its potential to identify important frames of videos.
Abstract (translated)
在本文中，我们探讨了使用基于Transformer的时空注意力网络（STAN）进行基于梯度的时序解释的可行性。首先，我们使用数据的全局和局部视图以及时间序列数据（即活动）的弱监督标签来训练STAN模型。然后，我们利用基于梯度的XAI技术（例如显着性图）来识别时间序列数据的显着帧。根据四个医学相关活动数据集的实验结果，STAN模型证明了其在识别视频重要帧方面的潜力。
URL
https://arxiv.org/abs/2405.17444
PDF
-
A Survey on Visual Mamba
2024-04-24 16:23:34Hanwei Zhang, Ying Zhu, Dan Wang, Lijun Zhang, Tianxiang Chen, Zi YeAbstract
State space models (SSMs) with selection mechanisms and hardware-aware architectures, namely Mamba, have recently demonstrated significant promise in long-sequence modeling. Since the self-attention mechanism in transformers has quadratic complexity with image size and increasing computational demands, the researchers are now exploring how to adapt Mamba for computer vision tasks. This paper is the first comprehensive survey aiming to provide an in-depth analysis of Mamba models in the field of computer vision. It begins by exploring the foundational concepts contributing to Mamba's success, including the state space model framework, selection mechanisms, and hardware-aware design. Next, we review these vision mamba models by categorizing them into foundational ones and enhancing them with techniques such as convolution, recurrence, and attention to improve their sophistication. We further delve into the widespread applications of Mamba in vision tasks, which include their use as a backbone in various levels of vision processing. This encompasses general visual tasks, Medical visual tasks (e.g., 2D / 3D segmentation, classification, and image registration, etc.), and Remote Sensing visual tasks. We specially introduce general visual tasks from two levels: High/Mid-level vision (e.g., Object detection, Segmentation, Video classification, etc.) and Low-level vision (e.g., Image super-resolution, Image restoration, Visual generation, etc.). We hope this endeavor will spark additional interest within the community to address current challenges and further apply Mamba models in computer vision.
Abstract (translated)
带有选择机制和硬件感知架构的状态空间模型（SSMs），如Mamba，在长序列建模方面最近取得了显著的进展。由于Transformer中自注意力机制的复杂性随着图像尺寸的增加而增加，计算机视觉任务的计算需求也在增加，因此研究人员现在正在探索如何将Mamba适应计算机视觉任务。本文是旨在为计算机视觉领域提供对Mamba模型的深入分析的第一篇全面调查。文章首先探讨了导致Mamba成功的基本概念，包括状态空间模型框架、选择机制和硬件感知设计。接下来，我们通过分类这些视觉Mamba模型为基本模型并使用卷积、递归和注意等技术对其进行改进，来回顾这些模型。我们深入探讨了Mamba在计算机视觉任务中的广泛应用，包括在各种级别视觉处理中的作为骨干的应用。这包括一般视觉任务（如物体检测、分割、分类和图像配准等）、医学视觉任务（如2D/3D分割、分类和图像配准等）和遥感视觉任务。我们特别引入了两个层面的通用视觉任务：高/中级别视觉（如物体检测、分割、视频分类等）和低级别视觉（如图像超分辨率、图像修复、视觉生成等）。我们希望这个努力将在社区中激发更多的兴趣，以解决当前的挑战并进一步将Mamba模型应用于计算机视觉。
URL
https://arxiv.org/abs/2404.15956
PDF
-
Learning Correlation Structures for Vision Transformers
2024-04-05 07:13:28Manjin Kim, Paul Hongsuck Seo, Cordelia Schmid, Minsu ChoAbstract
We introduce a new attention mechanism, dubbed structural self-attention (StructSA), that leverages rich correlation patterns naturally emerging in key-query interactions of attention. StructSA generates attention maps by recognizing space-time structures of key-query correlations via convolution and uses them to dynamically aggregate local contexts of value features. This effectively leverages rich structural patterns in images and videos such as scene layouts, object motion, and inter-object relations. Using StructSA as a main building block, we develop the structural vision transformer (StructViT) and evaluate its effectiveness on both image and video classification tasks, achieving state-of-the-art results on ImageNet-1K, Kinetics-400, Something-Something V1 & V2, Diving-48, and FineGym.
Abstract (translated)
我们提出了一种新的关注机制，被称为结构自注意（StructSA），它利用了注意力中自然出现的丰富相关模式。StructSA 通过卷积识别关键查询关联的空间时间结构，并使用它们动态地聚合价值特征的局部上下文。这有效地利用了图像和视频中的丰富结构模式，如场景布局、物体运动和物体之间的关系。使用StructSA作为主要构建模块，我们开发了结构视觉Transformer（StructViT），并在图像和视频分类任务上评估其效果，实现了ImageNet-1K、Kinetics-400、Something-Something V1&V2、Diving-48和FineGym的最好成绩。
URL
https://arxiv.org/abs/2404.03924
PDF
-
X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization
2024-03-28 19:45:35Anna Kukleva, Fadime Sener, Edoardo Remelli, Bugra Tekin, Eric Sauser, Bernt Schiele, Shugao MaAbstract
Lately, there has been growing interest in adapting vision-language models (VLMs) to image and third-person video classification due to their success in zero-shot recognition. However, the adaptation of these models to egocentric videos has been largely unexplored. To address this gap, we propose a simple yet effective cross-modal adaptation framework, which we call X-MIC. Using a video adapter, our pipeline learns to align frozen text embeddings to each egocentric video directly in the shared embedding space. Our novel adapter architecture retains and improves generalization of the pre-trained VLMs by disentangling learnable temporal modeling and frozen visual encoder. This results in an enhanced alignment of text embeddings to each egocentric video, leading to a significant improvement in cross-dataset generalization. We evaluate our approach on the Epic-Kitchens, Ego4D, and EGTEA datasets for fine-grained cross-dataset action generalization, demonstrating the effectiveness of our method. Code is available at this https URL
Abstract (translated)
近年来，由于其在零散拍摄识别方面的成功，人们对将视觉语言模型（VLMs）应用于图像和第三人称视频分类产生了浓厚兴趣。然而，将这些模型应用于以自我为中心的视频领域仍然是一个未被充分探索的领域。为了填补这一空白，我们提出了一个简单而有效的跨模态适应框架，我们称之为X-MIC。通过使用视频适配器，我们的管道学会了将冻结的文本嵌入对齐到每个以自我为中心的视频直接在共享嵌入空间中。我们新颖的适配器架构通过解开可学习的时间建模和冻结的视觉编码器的 learnable temporal modeling，保留并提高了预训练VLMs的泛化能力。这导致文本嵌入与每个以自我为中心的视频之间的增强对齐，从而在跨数据集通用性上取得了显著的提高。我们在Epic-Kitchens，Ego4D和EGTEA数据集上对细粒度跨数据集动作通用性进行评估，证明了我们的方法的的有效性。代码可以从该链接获得：
URL
https://arxiv.org/abs/2403.19811
PDF
-
Robustness and Visual Explanation for Black Box Image, Video, and ECG Signal Classification with Reinforcement Learning
2024-03-27 20:07:39Soumyendu Sarkar, Ashwin Ramesh Babu, Sajad Mousavi, Vineet Gundecha, Avisek Naug, Sahand GhorbanpourAbstract
We present a generic Reinforcement Learning (RL) framework optimized for crafting adversarial attacks on different model types spanning from ECG signal analysis (1D), image classification (2D), and video classification (3D). The framework focuses on identifying sensitive regions and inducing misclassifications with minimal distortions and various distortion types. The novel RL method outperforms state-of-the-art methods for all three applications, proving its efficiency. Our RL approach produces superior localization masks, enhancing interpretability for image classification and ECG analysis models. For applications such as ECG analysis, our platform highlights critical ECG segments for clinicians while ensuring resilience against prevalent distortions. This comprehensive tool aims to bolster both resilience with adversarial training and transparency across varied applications and data types.
Abstract (translated)
我们提出了一个通用的强化学习（RL）框架，针对不同模型类型的攻击进行优化，包括心电图信号分析（1D）、图像分类（2D）和视频分类（3D）。该框架专注于识别敏感区域，通过最小程度的扭曲和各种扭曲类型来诱导误分类。与最先进的 methods 相比，新RL 方法在所有三个应用中都表现出色，证明了其效率。我们的RL方法产生了卓越的局部化掩码，增强了图像分类和心电图分析模型的可解释性。对于心电图分析等应用，我们的平台在确保对常见扭曲的 resilience 同时，使临床医生关注的ECG段具有更高的鲁棒性。这一全面的工具旨在通过对抗性训练增强韧性和跨应用和数据类型的透明度。
URL
https://arxiv.org/abs/2403.18985
PDF
-
Leveraging Compressed Frame Sizes For Ultra-Fast Video Classification
2024-03-13 14:35:13Yuxing Han, Yunan Ding, Chen Ye Gan, Jiangtao WenAbstract
Classifying videos into distinct categories, such as Sport and Music Video, is crucial for multimedia understanding and retrieval, especially when an immense volume of video content is being constantly generated. Traditional methods require video decompression to extract pixel-level features like color, texture, and motion, thereby increasing computational and storage demands. Moreover, these methods often suffer from performance degradation in low-quality videos. We present a novel approach that examines only the post-compression bitstream of a video to perform classification, eliminating the need for bitstream decoding. To validate our approach, we built a comprehensive data set comprising over 29,000 YouTube video clips, totaling 6,000 hours and spanning 11 distinct categories. Our evaluations indicate precision, accuracy, and recall rates consistently above 80%, many exceeding 90%, and some reaching 99%. The algorithm operates approximately 15,000 times faster than real-time for 30fps videos, outperforming traditional Dynamic Time Warping (DTW) algorithm by seven orders of magnitude.
Abstract (translated)
将视频分类为不同的类别（如体育和音乐视频）对于多媒体理解和检索至关重要，尤其是在生成大量视频内容的情况下。传统方法需要对视频进行解压缩以提取像素级别的特征，如颜色、纹理和运动，从而增加计算和存储需求。此外，这些方法在低质量视频中常常存在性能退化。我们提出了一种新方法，只检查视频的后压缩比特流进行分类，无需进行比特流解码。为了验证我们的方法，我们构建了一个由29,000个YouTube视频片段组成的全集，总计6,000小时，跨越11个类别的综合数据集。我们的评估结果表明，精度、准确性和召回率均高于80%，许多甚至超过90%，有些达到99%。该算法在30fps视频上的运行速度约为实时速度的15,000倍，比传统动态时间膨胀（DTW）算法快7 orders of magnitude。
URL
https://arxiv.org/abs/2403.08580
PDF
-
Pig aggression classification using CNN, Transformers and Recurrent Networks
2024-03-13 13:38:58Junior Silva Souza, Eduardo Bedin, Gabriel Toshio Hirokawa Higa, Newton Loebens, Hemerson PistoriAbstract
The development of techniques that can be used to analyze and detect animal behavior is a crucial activity for the livestock sector, as it is possible to monitor the stress and animal welfare and contributes to decision making in the farm. Thus, the development of applications can assist breeders in making decisions to improve production performance and reduce costs, once the animal behavior is analyzed by humans and this can lead to susceptible errors and time consumption. Aggressiveness in pigs is an example of behavior that is studied to reduce its impact through animal classification and identification. However, this process is laborious and susceptible to errors, which can be reduced through automation by visually classifying videos captured in controlled environment. The captured videos can be used for training and, as a result, for classification through computer vision and artificial intelligence, employing neural network techniques. The main techniques utilized in this study are variants of transformers: STAM, TimeSformer, and ViViT, as well as techniques using convolutions, such as ResNet3D2, Resnet(2+1)D, and CnnLstm. These techniques were employed for pig video classification with the objective of identifying aggressive and non-aggressive behaviors. In this work, various techniques were compared to analyze the contribution of using transformers, in addition to the effectiveness of the convolution technique in video classification. The performance was evaluated using accuracy, precision, and recall. The TimerSformer technique showed the best results in video classification, with median accuracy of 0.729.
Abstract (translated)
用于分析和检测动物行为的技术的开发对于畜牧业至关重要，因为这可以帮助监测应激和动物福利，并在农场决策中做出贡献。因此，应用程序的开发有助于育种者根据人类分析动物行为后做出改进生产表现和降低成本的决定。通过视觉分类控制环境捕获的视频可以减少其影响。然而，这个过程费力且容易出错，可以通过自动化视觉分类视频来减少错误。捕获的视频可以用于培训，进而通过计算机视觉和人工智能进行分类，采用神经网络技术。 在本研究中使用的技术包括变体的Transformer：STAM，TimeSformer和ViViT，以及使用卷积的技术，如ResNet3D2，Resnet（2+1）D和CnnLstm。这些技术用于将视频分类为攻击性和非攻击性行为。在本研究中，还比较了使用Transformer和卷积技术在视频分类中的效果。STAM技术在视频分类中表现最佳，其均方误差（MSE）为0.729。
URL
https://arxiv.org/abs/2403.08528
PDF
-
Learning Expressive And Generalizable Motion Features For Face Forgery Detection
2024-03-08 09:25:48Jingyi Zhang, Peng Zhang, Jingjing Wang, Di Xie, Shiliang PuAbstract
Previous face forgery detection methods mainly focus on appearance features, which may be easily attacked by sophisticated manipulation. Considering the majority of current face manipulation methods generate fake faces based on a single frame, which do not take frame consistency and coordination into consideration, artifacts on frame sequences are more effective for face forgery detection. However, current sequence-based face forgery detection methods use general video classification networks directly, which discard the special and discriminative motion information for face manipulation detection. To this end, we propose an effective sequence-based forgery detection framework based on an existing video classification method. To make the motion features more expressive for manipulation detection, we propose an alternative motion consistency block instead of the original motion features module. To make the learned features more generalizable, we propose an auxiliary anomaly detection block. With these two specially designed improvements, we make a general video classification network achieve promising results on three popular face forgery datasets.
Abstract (translated)
过去的面部伪造检测方法主要关注外观特征，这可能很容易被高级操纵所攻击。考虑到当前大多数面部 manipulation 方法都是基于单个帧生成的假脸，没有考虑帧的一致性和协调性，序列中的伪影对于面部伪造检测来说更为有效。然而，现有的序列基于面部伪造检测的方法直接使用通用视频分类网络，这忽略了面部操纵检测的特殊和鉴别信息。为此，我们提出了一个基于现有视频分类方法的序列基于伪造检测框架。为了使操纵检测更具有表现力，我们提出了一个替代的动态一致性模块，而不是原始动态特征模块。为了使学习到的特征更具通用性，我们提出了一个辅助异常检测模块。通过这两个特别设计的改进，我们使一般视频分类网络在三个流行的面部伪造数据集上取得了良好的结果。
URL
https://arxiv.org/abs/2403.05172
PDF
-
A Multimodal Handover Failure Detection Dataset and Baselines
2024-02-28 13:29:28Santosh Thoduka, Nico Hochgeschwender, Juergen Gall, Paul G. PlögerAbstract
An object handover between a robot and a human is a coordinated action which is prone to failure for reasons such as miscommunication, incorrect actions and unexpected object properties. Existing works on handover failure detection and prevention focus on preventing failures due to object slip or external disturbances. However, there is a lack of datasets and evaluation methods that consider unpreventable failures caused by the human participant. To address this deficit, we present the multimodal Handover Failure Detection dataset, which consists of failures induced by the human participant, such as ignoring the robot or not releasing the object. We also present two baseline methods for handover failure detection: (i) a video classification method using 3D CNNs and (ii) a temporal action segmentation approach which jointly classifies the human action, robot action and overall outcome of the action. The results show that video is an important modality, but using force-torque data and gripper position help improve failure detection and action segmentation accuracy.
Abstract (translated)
机器人与人类之间的物体传递是一个协调的动作,由于诸如沟通不畅、操作不正确或意外物体属性等原因,容易出现失败。现有关于物体传递失败检测和预防的作品主要集中在由于物体滑落或外部干扰导致的故障的预防上。然而,目前缺乏考虑人类参与者无法预防的故障的数据集和评估方法。为了弥补这一不足,我们提出了多模态物体传递失败检测数据集,其中包括由人类参与者引起的事故,例如忽略机器人或未释放物体。我们还提出了两种基本的物体传递失败检测方法:(i)使用3D CNN的视觉分类方法;(ii)一种将人类动作、机器人动作和动作结果共同分类为时序动作的方法。结果表明,视频是一个重要的模式,但使用力和扭矩数据以及抓爪位置能够提高故障检测和动作分割的准确性。
URL
https://arxiv.org/abs/2402.18319
PDF
-
Video Annotator: A framework for efficiently building video classifiers using vision-language models and active learning
2024-02-09 17:19:05Amir Ziai, Aneesh VartakaviAbstract
High-quality and consistent annotations are fundamental to the successful development of robust machine learning models. Traditional data annotation methods are resource-intensive and inefficient, often leading to a reliance on third-party annotators who are not the domain experts. Hard samples, which are usually the most informative for model training, tend to be difficult to label accurately and consistently without business context. These can arise unpredictably during the annotation process, requiring a variable number of iterations and rounds of feedback, leading to unforeseen expenses and time commitments to guarantee quality. We posit that more direct involvement of domain experts, using a human-in-the-loop system, can resolve many of these practical challenges. We propose a novel framework we call Video Annotator (VA) for annotating, managing, and iterating on video classification datasets. Our approach offers a new paradigm for an end-user-centered model development process, enhancing the efficiency, usability, and effectiveness of video classifiers. Uniquely, VA allows for a continuous annotation process, seamlessly integrating data collection and model training. We leverage the zero-shot capabilities of vision-language foundation models combined with active learning techniques, and demonstrate that VA enables the efficient creation of high-quality models. VA achieves a median 6.8 point improvement in Average Precision relative to the most competitive baseline across a wide-ranging assortment of tasks. We release a dataset with 153k labels across 56 video understanding tasks annotated by three professional video editors using VA, and also release code to replicate our experiments at: this http URL.
Abstract (translated)
高品质和一致的注释是成功开发稳健的机器学习模型的基础。传统的数据注释方法资源密集且效率低下，通常导致在领域专家不是的情况下依赖第三方注释者。难样本（通常是模型训练中最有信息量的）往往在无商业上下文的情况下难以准确且一致地标注。这些挑战在注释过程中可能会意外出现，需要进行反复的迭代和反馈，导致未知的开销和保证质量的时间要求。我们提出了一种名为视频注释者（VA）的新框架，用于标注、管理和迭代视频分类数据集。我们的方法为以用户为中心的模型开发过程提供了一个新的范例，提高了视频分类器的效率、可用性和效果。值得注意的是，VA允许进行连续的注释过程，将数据收集和模型训练无缝集成。我们结合了视觉语言基础模型的零 shot功能与积极学习技术，证明了VA能够高效地创建高质量的模型。VA在广泛的任务中实现了平均6.8个点的AP提高，与最具有竞争力的基线相比。我们发布了由三个专业视频编辑师标注的56个视频理解任务的数据集，并在此处发布代码以复制我们的实验：http:// this URL.
URL
https://arxiv.org/abs/2402.06560
PDF
-
Time-, Memory- and Parameter-Efficient Visual Adaptation
2024-02-05 10:55:47Otniel-Bogdan Mercea, Alexey Gritsenko, Cordelia Schmid, Anurag ArnabAbstract
As foundation models become more popular, there is a growing need to efficiently finetune them for downstream tasks. Although numerous adaptation methods have been proposed, they are designed to be efficient only in terms of how many parameters are trained. They, however, typically still require backpropagating gradients throughout the model, meaning that their training-time and -memory cost does not reduce as significantly. We propose an adaptation method which does not backpropagate gradients through the backbone. We achieve this by designing a lightweight network in parallel that operates on features from the frozen, pretrained backbone. As a result, our method is efficient not only in terms of parameters, but also in training-time and memory usage. Our approach achieves state-of-the-art accuracy-parameter trade-offs on the popular VTAB benchmark, and we further show how we outperform prior works with respect to training-time and -memory usage too. We further demonstrate the training efficiency and scalability of our method by adapting a vision transformer backbone of 4 billion parameters for the computationally demanding task of video classification, without any intricate model parallelism. Here, we outperform a prior adaptor-based method which could only scale to a 1 billion parameter backbone, or fully-finetuning a smaller backbone, with the same GPU and less training time.
Abstract (translated)
作为基础模型越来越受欢迎，对于下游任务进行有效微调的需求不断增加。虽然已经提出了许多自适应方法，但它们的设计仅在参数训练方面有效。然而，它们通常还需要在模型中进行反向传播梯度，这意味着它们的训练时间和内存成本不会显著降低。我们提出了一种不通过骨干网络反向传播梯度的自适应方法。我们通过设计一个轻量级的并行网络来实现这个目标，该网络操作于预训练骨架中的特征。这样，我们的方法不仅在参数方面有效，而且在训练时间和内存使用方面也有效。我们的方法在热门的VTAB基准上实现了与最先进方法相同的准确率参数权衡，并且我们还进一步证明了我们在训练时间和内存使用方面的优势。为了进一步证明我们方法的训练效率和可扩展性，我们将一个具有40亿参数的视觉Transformer骨干网络调整为用于计算密集型视频分类任务的模型，而没有任何复杂的模型并行。在这里，我们超过了基于先前的自适应方法，该方法只能扩展到10亿参数的骨干网络，或者对较小的骨干网络进行完全微调，使用相同的GPU和更短的时间进行训练。
URL
https://arxiv.org/abs/2402.02887
PDF
-
Short-Form Videos and Mental Health: A Knowledge-Guided Multimodal Neural Topic Model
2024-01-11 03:36:47Jiaheng Xie, Ruicheng Liang, Yidong Chai, Yang LiuAbstract
While short-form videos head to reshape the entire social media landscape, experts are exceedingly worried about their depressive impacts on viewers, as evidenced by medical studies. To prevent widespread consequences, platforms are eager to predict these videos' impact on viewers' mental health. Subsequently, they can take intervention measures, such as revising recommendation algorithms and displaying viewer discretion. Nevertheless, applicable predictive methods lack relevance to well-established medical knowledge, which outlines clinically proven external and environmental factors of depression. To account for such medical knowledge, we resort to an emergent methodological discipline, seeded Neural Topic Models (NTMs). However, existing seeded NTMs suffer from the limitations of single-origin topics, unknown topic sources, unclear seed supervision, and suboptimal convergence. To address those challenges, we develop a novel Knowledge-guided Multimodal NTM to predict a short-form video's depressive impact on viewers. Extensive empirical analyses using TikTok and Douyin datasets prove that our method outperforms state-of-the-art benchmarks. Our method also discovers medically relevant topics from videos that are linked to depressive impact. We contribute to IS with a novel video analytics method that is generalizable to other video classification problems. Practically, our method can help platforms understand videos' mental impacts, thus adjusting recommendations and video topic disclosure.
Abstract (translated)
尽管短时视频正在彻底改变社交媒体格局，但专家对它们对观众产生的沮丧影响感到担忧，这是医学研究的结果。为了防止广泛后果，平台渴望预测这些视频对观众心理健康的影响。因此，他们可以采取干预措施，例如修改推荐算法和显示观众自主性。然而，适用的预测方法与已建立的抑郁症医学知识不相关，这揭示了抑郁症的外部和环境因素。为了应对这种医学知识，我们求助于新兴的方法论学科，即以知识为导向的神经主题模型（NTMs）。然而，现有的NTM存在单一来源主题、未知主题来源、不明确的种子监督和次优收敛等限制。为了应对这些挑战，我们开发了一种新知识引导的多模态NTM，以预测短时视频对观众的沮丧影响。通过使用TikTok和Douyin数据集的丰富实证分析证明，我们的方法超越了最先进的基准。我们的方法还从与沮丧影响相关的视频中发现了具有医学相关性的主题。我们与IS合作研发了一种可应用于其他视频分类问题的视频分析方法。实际上，我们的方法可以帮助平台理解视频的心理健康影响，从而调整推荐和视频主题披露。
URL
https://arxiv.org/abs/2402.10045
PDF
