Abstract
Video action localization aims to find timings of a specific action from a long video. Although existing learning-based approaches have been successful, those require annotating videos that come with a considerable labor cost. This paper proposes a learning-free, open-vocabulary approach based on emerging vision-language models (VLM). The challenge stems from the fact that VLMs are neither designed to process long videos nor tailored for finding actions. We overcome these problems by extending an iterative visual prompting technique. Specifically, we sample video frames into a concatenated image with frame index labels, making a VLM guess a frame that is considered to be closest to the start/end of the action. Iterating this process by narrowing a sampling time window results in finding a specific frame of start and end of an action. We demonstrate that this sampling technique yields reasonable results, illustrating a practical extension of VLMs for understanding videos.
Abstract (translated)
视频动作定位旨在从长视频中查找特定动作的时间。虽然现有的基于学习的 approaches 已经取得了成功，但它们需要标注带有相当大劳动成本的视频。本文提出了一种基于新兴的视觉语言模型（VLM）的学习无，开卷积文本文提出的技术基于 emerging vision-language models (VLM)。挑战来自于 VLMs 既没有设计来处理长视频，也不是为了找到动作而设计的。我们通过扩展迭代视觉提示技术来克服这些问题。具体来说，我们将视频帧采样到与帧编号标签连接的 concatenated 图像中，使 VLM 猜测一个被认为是动作开始/结束的帧。通过缩小采样时间窗口，迭代这个过程可以找到特定动作的开始和结束。我们证明了这种采样技术产生了合理的结果，证明了 VLMs 在理解视频方面具有实际扩展。
URL
https://arxiv.org/abs/2408.17422
