Abstract
Understanding how humans and artificial intelligence systems process complex narrative videos is a fundamental challenge at the intersection of neuroscience and machine learning. This study investigates how the temporal context length of video clips (3--12 s clips) and the narrative-task prompting shape brain-model alignment during naturalistic movie watching. Using fMRI recordings from participants viewing full-length movies, we examine how brain regions sensitive to narrative context dynamically represent information over varying timescales and how these neural patterns align with model-derived features. We find that increasing clip duration substantially improves brain alignment for multimodal large language models (MLLMs), whereas unimodal video models show little to no gain. Further, shorter temporal windows align with perceptual and early language regions, while longer windows preferentially align higher-order integrative regions, mirrored by a layer-to-cortex hierarchy in MLLMs. Finally, narrative-task prompts (multi-scene summary, narrative summary, character motivation, and event boundary detection) elicit task-specific, region-dependent brain alignment patterns and context-dependent shifts in clip-level tuning in higher-order regions. Together, our results position long-form narrative movies as a principled testbed for probing biologically relevant temporal integration and interpretable representations in long-context MLLMs.
Abstract (translated)
理解人类和人工智能系统如何处理复杂的叙事视频是神经科学与机器学习交叉领域中的一个基本挑战。本研究探讨了视频片段的时间上下文长度（3至12秒）以及叙述任务提示如何在观看自然主义电影时影响大脑模型的对齐情况。通过从参与者观看全长电影期间采集的功能磁共振成像（fMRI）记录，我们考察了哪些脑区在面对不同的时间尺度上是如何动态地表示叙事信息的，以及这些神经模式与由机器学习模型推导出的特征如何匹配。 研究发现，延长视频片段的时间可以显著提升多模态大型语言模型（MLLMs）的大脑对齐度，而单模态视频模型则几乎没有改善。此外，较短的时间窗口倾向于在感知和早期语言区域中找到对应，而较长的时间窗口则偏好于更高层次的综合区域，这一现象与MLLMs中的层级-皮层层次结构相呼应。 最后，叙事任务提示（如跨场景摘要、叙述性总结、人物动机分析以及事件边界检测）激发了特定的任务相关的脑区对齐模式，并在较高阶区域内引发了依据上下文而变化的片段级别调节。综上所述，我们的研究结果将长篇叙事电影定位为一种探查生物相关时间整合和多上下文模型解释能力的理想实验平台。
URL
https://arxiv.org/abs/2602.07570
