Abstract
Pre-trained vision-language models (VLMs) have enabled significant progress in open vocabulary computer vision tasks such as image classification, object detection and image segmentation. Some recent works have focused on extending VLMs to open vocabulary single label action classification in videos. However, previous methods fall short in holistic video understanding which requires the ability to simultaneously recognize multiple actions and entities e.g., objects in the video in an open vocabulary setting. We formulate this problem as open vocabulary multilabel video classification and propose a method to adapt a pre-trained VLM such as CLIP to solve this task. We leverage large language models (LLMs) to provide semantic guidance to the VLM about class labels to improve its open vocabulary performance with two key contributions. First, we propose an end-to-end trainable architecture that learns to prompt an LLM to generate soft attributes for the CLIP text-encoder to enable it to recognize novel classes. Second, we integrate a temporal modeling module into CLIP's vision encoder to effectively model the spatio-temporal dynamics of video concepts as well as propose a novel regularized finetuning technique to ensure strong open vocabulary classification performance in the video domain. Our extensive experimentation showcases the efficacy of our approach on multiple benchmark datasets.
Abstract (translated)
预训练的视觉-语言模型（VLMs）已经在诸如图像分类、目标检测和图像分割等开放词汇计算机视觉任务中取得了显著的进展。一些最近的工作将重点放在将VLMs扩展到开放词汇单标签动作分类在视频上。然而，之前的方法在全面理解视频方面存在缺陷，这需要同时识别多个动作和实体，例如视频中的对象。我们将这个问题转化为开放词汇多标签视频分类，并提出了将预训练的VLM如CLIP适应这种任务的方法。我们利用大型语言模型（LLMs）为VLM提供语义指导，以便提高其开放词汇性能。我们的方法具有两个关键贡献：第一，我们提出了一个端到端的训练架构，学习提示LLM生成软特征，以使它能够识别新的类别。第二，我们将一个时间建模模块集成到CLIP的视觉编码器中，有效地建模了视频概念的空间和时间动态，并提出了一个新的正则化微调技术，以确保在视频域中具有强大的开放词汇分类性能。我们对多个基准数据集的实验表明，我们的方法在多个方面都具有有效性。
URL
https://arxiv.org/abs/2407.09073
