Abstract
Despite recent progress in video large language models (VideoLLMs), a key open challenge remains: how to equip models with chain-of-thought (CoT) reasoning abilities grounded in fine-grained object-level video understanding. Existing instruction-tuned models, such as the Qwen and LLaVA series, are trained on high-level video-text pairs, often lacking structured annotations necessary for compositional, step-by-step reasoning. We propose CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks, a new framework that decomposes complex video questions of existing datasets (e.g., NeXT-QA, STAR) into four entity-level foundational tasks: frame localization, entity tracking, spatial and temporal relation extraction. By embedding these intermediate CoT-style reasoning steps into the input, CoTasks enables models to explicitly perform object-centric spatiotemporal reasoning. Experiments on the NeXT-QA benchmark show that CoTasks significantly enhance inference performance: LLaVA-video-7B improves by +3.3 points in average GPT-4 evaluation score, and Qwen2.5-VL-3B gains +17.4, with large boosts in causal (+14.6), temporal (+10.9), and descriptive (+48.1) subcategories. These results demonstrate the effectiveness of CoTasks as a structured CoT-style supervision framework for improving compositional video reasoning.
Abstract (translated)
尽管在视频大型语言模型（VideoLLM）方面取得了最近的进展，但一个关键的开放性挑战仍然存在：如何使这些模型具备基于细粒度对象级别视频理解的链式思维（CoT）推理能力。现有的指令微调模型，如Qwen和LLaVA系列，是基于高层次的视频文本对进行训练的，通常缺少组成性和分步骤推理所需的结构化注释。我们提出了CoTasks：一种基于链式思维的视频指令微调任务框架，它将现有数据集（例如NeXT-QA、STAR）中的复杂视频问题分解为四个实体级别的基础任务：帧定位、对象跟踪、空间和时间关系抽取。通过在输入中嵌入这些中间的CoT风格推理步骤，CoTasks使模型能够明确地执行以对象为中心的空间-时间推理。 在NeXT-QA基准上的实验表明，CoTasks显著提高了推理性能：LLaVA-video-7B在平均GPT-4评估分数上提升了+3.3分，而Qwen2.5-VL-3B则提升了+17.4分，在因果（+14.6）、时间（+10.9）和描述性（+48.1）子类别中取得了显著的进步。这些结果证明了CoTasks作为结构化的CoT风格监督框架在改进组合视频推理方面的有效性。
URL
https://arxiv.org/abs/2507.13609
