Abstract
The advancements in large language models (LLMs) have propelled the improvement of video understanding tasks by incorporating LLMs with visual models. However, most existing LLM-based models (e.g., VideoLLaMA, VideoChat) are constrained to processing short-duration videos. Recent attempts to understand long-term videos by extracting and compressing visual features into a fixed memory size. Nevertheless, those methods leverage only visual modality to merge video tokens and overlook the correlation between visual and textual queries, leading to difficulties in effectively handling complex question-answering tasks. To address the challenges of long videos and complex prompts, we propose AdaCM$^2$, which, for the first time, introduces an adaptive cross-modality memory reduction approach to video-text alignment in an auto-regressive manner on video streams. Our extensive experiments on various video understanding tasks, such as video captioning, video question answering, and video classification, demonstrate that AdaCM$^2$ achieves state-of-the-art performance across multiple datasets while significantly reducing memory usage. Notably, it achieves a 4.5% improvement across multiple tasks in the LVU dataset with a GPU memory consumption reduction of up to 65%.
Abstract (translated)
大型语言模型（LLMs）的进步通过将这些模型与视觉模型相结合，推动了视频理解任务的改进。然而，大多数现有的基于LLM的模型（例如VideoLLaMA、VideoChat）仅限于处理短时长视频。最近尝试通过提取和压缩视觉特征到固定内存大小来理解长期视频。尽管如此，这些方法只利用视觉模态来合并视频令牌，并忽视了视觉与文本查询之间的关联性，导致难以有效地处理复杂的问答任务。为了解决长时间视频和复杂提示的挑战，我们提出了AdaCM$^2$，该模型首次引入了一种自适应跨模态记忆缩减方法，在自动回归方式下应用于视频流中的视频-文本对齐。我们在各种视频理解任务上的广泛实验，如视频字幕生成、视频问答和视频分类，表明AdaCM$^2$在多个数据集上达到了最先进的性能，并显著减少了内存使用量。值得注意的是，在LVU数据集中，它实现了跨多项任务4.5%的提升，同时GPU内存消耗减少高达65%。
URL
https://arxiv.org/abs/2411.12593
