Abstract
As the scale of data and models for video understanding rapidly expand, handling long-form video input in transformer-based models presents a practical challenge. Rather than resorting to input sampling or token dropping, which may result in information loss, token merging shows promising results when used in collaboration with transformers. However, the application of token merging for long-form video processing is not trivial. We begin with the premise that token merging should not rely solely on the similarity of video tokens; the saliency of tokens should also be considered. To address this, we explore various video token merging strategies for long-form video classification, starting with a simple extension of image token merging, moving to region-concentrated merging, and finally proposing a learnable video token merging (VTM) algorithm that dynamically merges tokens based on their saliency. Extensive experimental results show that we achieve better or comparable performances on the LVU, COIN, and Breakfast datasets. Moreover, our approach significantly reduces memory costs by 84% and boosts throughput by approximately 6.89 times compared to baseline algorithms.
Abstract (translated)
随着视频理解中数据和模型规模的迅速扩大，基于变压器模型处理长格式视频输入变得具有实际挑战性。与其依赖可能导致信息丢失的输入采样或标记丢弃，与变压器协作使用的标记合并显示出有希望的结果。然而，对长格式视频处理应用标记合并并不简单。我们从这样一个前提开始：标记合并不应仅依靠视频标记之间的相似性；还应考虑标记的重要性。为了解决这一问题，我们探索了多种针对长格式视频分类的视频标记合并策略，从图像标记合并的简单扩展入手，转向集中于区域的合并方法，最终提出了一种基于可学习的视频标记合并（VTM）算法，该算法根据标记的重要性动态地进行合并。广泛的实验结果表明，在LVU、COIN和Breakfast数据集上我们取得了更好的或相当的表现。此外，与基线算法相比，我们的方法显著降低了84%的内存成本，并将吞吐量提高了大约6.89倍。
URL
https://arxiv.org/abs/2410.23782
