Abstract
Narrative videos, such as movies, pose significant challenges in video understanding due to their rich contexts (characters, dialogues, storylines) and diverse demands (identify who, relationship, and reason). In this paper, we introduce MovieSeq, a multimodal language model developed to address the wide range of challenges in understanding video contexts. Our core idea is to represent videos as interleaved multimodal sequences (including images, plots, videos, and subtitles), either by linking external knowledge databases or using offline models (such as whisper for subtitles). Through instruction-tuning, this approach empowers the language model to interact with videos using interleaved multimodal instructions. For example, instead of solely relying on video as input, we jointly provide character photos alongside their names and dialogues, allowing the model to associate these elements and generate more comprehensive responses. To demonstrate its effectiveness, we validate MovieSeq's performance on six datasets (LVU, MAD, Movienet, CMD, TVC, MovieQA) across five settings (video classification, audio description, video-text retrieval, video captioning, and video question-answering). The code will be public at this https URL.
Abstract (translated)
叙事视频（如电影）在视频理解方面面临着显著的挑战，因为它们具有丰富的上下文（角色、对话、故事情节）和多样化的要求（确定谁、关系和原因）。在本文中，我们介绍了MovieSeq，一种多模态语言模型，旨在解决理解视频上下文的广泛范围挑战。我们的核心思想是将视频表示为交织的多模态序列（包括图像、剧情、视频和字幕），或者通过链接外部知识数据库或使用离线模型（如whisper for subtitles）来实现。通过指令调整，这种方法使语言模型能够使用交织的多模态指令与视频进行交互。例如， instead of solely relying on video as input, we jointly provide character photos alongside their names and dialogues, allowing the model to associate these elements and generate more comprehensive responses. 为了证明其有效性，我们在六个数据集（LVU，MAD，Movienet，CMD，TVC，MovieQA）和五个设置（视频分类、音频描述、视频-文本检索、视频字幕、视频问题回答）上对MovieSeq进行了评估。代码将在这个https URL上公开。
URL
https://arxiv.org/abs/2407.21757
