Video_Caption
-
VSAS-BENCH: Real-Time Evaluation of Visual Streaming Assistant Models
2026-04-08 22:31:20Pavan Kumar Anasosalu Vasu, Cem Koc, Fartash Faghri, Chun-Liang Li, Bo Feng, Zhengfeng Lai, Meng Cao, Oncel Tuzel, Hadi PouransariAbstract
Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames. This is a core mechanism for real-time visual assistants. Existing VLM frameworks predominantly assess models in offline settings. In contrast, the performance of a streaming VLM depends on additional metrics beyond pure video understanding, including proactiveness, which reflects the timeliness of the model's responses, and consistency, which captures the robustness of its responses over time. To address this limitation, we propose VSAS-Bench, a new framework and benchmark for Visual Streaming Assistants. In contrast to prior benchmarks that primarily employ single-turn question answering on video inputs, VSAS-Bench features temporally dense annotations with over 18,000 annotations across diverse input domains and task types. We introduce standardized synchronous and asynchronous evaluation protocols, along with metrics that isolate and measure distinct capabilities of streaming VLMs. Using this framework, we conduct large-scale evaluations of recent video and streaming VLMs, analyzing the accuracy-latency trade-off under key design factors such as memory buffer length, memory access policy, and input resolution, yielding several practical insights. Finally, we show empirically that conventional VLMs can be adapted to streaming settings without additional training, and demonstrate that these adapted models outperform recent streaming VLMs. For example, Qwen3-VL-4B surpasses Dispider, the best streaming VLM on our benchmark, by 3% under the asynchronous protocol. The benchmark and code will be available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.07634
PDF
-
See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs
2026-04-07 09:54:33Yicheng Ji, Jun Zhang, Jinpeng Chen, Cong Wang, Lidan Shou, Gang Chen, Huan LiAbstract
Video Large Language Models (Video-LLMs) excel in video understanding but suffer from high inference latency during autoregressive generation. Speculative Decoding (SD) mitigates this by applying a draft-and-verify paradigm, yet existing methods are constrained by rigid exact-match rules, severely limiting the acceleration potential. To bridge this gap, we propose LVSpec, the first training-free loosely SD framework tailored for Video-LLMs. Grounded in the insight that generation is governed by sparse visual-relevant anchors (mandating strictness) amidst abundant visual-irrelevant fillers (permitting loose verification), LVSpec employs a lightweight visual-relevant token identification scheme to accurately pinpoint the former. To further maximize acceptance, we augment this with a position-shift tolerant mechanism that effectively salvages positionally mismatched but semantically equivalent tokens. Experiments demonstrate that LVSpec achieves high fidelity and speed: it preserves >99.8 of target performance while accelerating Qwen2.5-VL-32B by 2.70x and LLaVA-OneVision-72B by 2.94x. Notably, it boosts the mean accepted length and speedup ratio by 136% and 35% compared to SOTA training-free SD methods for Video-LLMs.
Abstract (translated)
URL
https://arxiv.org/abs/2604.05650
PDF
-
Watch Before You Answer: Learning from Visually Grounded Post-Training
2026-04-06 19:22:48Yuxuan Zhang, EunJeong Hwang, Huaisong Zhang, Penghui Du, Yiming Jia, Dongfu Jiang, Xuan He, Shenhui Zhang, Ping Nie, Peter West, Kelsey R. AllenAbstract
It is critical for vision-language models (VLMs) to comprehensively understand visual, temporal, and textual cues. However, despite rapid progress in multimodal modeling, video understanding performance still lags behind text-based reasoning. In this work, we find that progress is even worse than previously assumed: commonly reported long video understanding benchmarks contain 40-60% of questions that can be answered using text cues alone. Furthermore, we find that these issues are also pervasive in widely used post-training datasets, potentially undercutting the ability of post-training to improve VLM video understanding performance. Guided by this observation, we introduce VidGround as a simple yet effective solution: using only the actual visually grounded questions without any linguistic biases for post-training. When used in tandem with RL-based post-training algorithms, this simple technique improves performance by up to 6.2 points relative to using the full dataset, while using only 69.1% of the original post-training data. Moreover, we show that data curation with a simple post-training algorithm outperforms several more complex post-training techniques, highlighting that data quality is a major bottleneck for improving video understanding in VLMs. These results underscore the importance of curating post-training data and evaluation benchmarks that truly require visual grounding to advance the development of more capable VLMs. Project page: this http URL.
Abstract (translated)
URL
https://arxiv.org/abs/2604.05117
PDF
-
SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaboration
2026-04-06 18:30:50Zhongyu Yang, Zuhao Yang, Shuo Zhan, Tan Yue, Wei Pang, Yingfang YuanAbstract
Video question answering (VideoQA) is a challenging task that requires integrating spatial, temporal, and semantic information to capture the complex dynamics of video sequences. Although recent advances have introduced various approaches for video understanding, most existing methods still rely on locating relevant frames to answer questions rather than reasoning through the evolving storyline as humans do. Humans naturally interpret videos through coherent storylines, an ability that is crucial for making robust and contextually grounded predictions. To address this gap, we propose SVAgent, a storyline-guided cross-modal multi-agent framework for VideoQA. The storyline agent progressively constructs a narrative representation based on frames suggested by a refinement suggestion agent that analyzes historical failures. In addition, cross-modal decision agents independently predict answers from visual and textual modalities under the guidance of the evolving storyline. Their outputs are then evaluated by a meta-agent to align cross-modal predictions and enhance reasoning robustness and answer consistency. Experimental results demonstrate that SVAgent achieves superior performance and interpretability by emulating human-like storyline reasoning in video understanding.
Abstract (translated)
视频问答（VideoQA）是一项具有挑战性的任务，需要整合空间、时序和语义信息以捕捉视频序列的复杂动态。尽管近期研究已提出多种视频理解方法，但现有技术仍主要依赖定位相关帧来回答问题，而非像人类一样通过演变的故事情节进行推理。人类天然通过连贯的叙事来解读视频，这种能力对于做出稳健且基于上下文的预测至关重要。为弥补这一差距，我们提出了SVAgent——一个叙事引导的跨模态多智能体视频问答框架。其中，叙事智能体基于由分析历史失败案例的优化建议智能体所推荐的帧，逐步构建叙事表征；同时，跨模态决策智能体在演变的叙事引导下，分别从视觉和文本模态独立预测答案。随后，元智能体评估这些输出以对齐跨模态预测，并增强推理稳健性与答案一致性。实验结果表明，SVAgent通过模拟人类式的叙事推理，在视频理解中实现了卓越性能和可解释性。
URL
https://arxiv.org/abs/2604.05079
PDF
-
Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
2026-04-06 17:59:56Chaoyou Fu, Haozhi Yuan, Yuhao Dong, Yi-Fan Zhang, Yunhang Shen, Xiaoxing Hu, Xueying Li, Jinsen Su, Chengwu Long, Xiaoyao Xie, Yongkang Xie, Xiawu Zheng, Xue Yang, Haoyu Cao, Yunsheng Wu, Ziwei Liu, Xing Sun, Caifeng Shan, Ran HeAbstract
With the rapid advancement of video understanding, existing benchmarks are becoming increasingly saturated, exposing a critical discrepancy between inflated leaderboard scores and real-world model capabilities. To address this widening gap, we introduce Video-MME-v2, a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding. To systematically evaluate model capabilities, we design a \textbf{progressive tri-level hierarchy} that incrementally increases the complexity of video comprehension, ranging from multi-point visual information aggregation, to temporal dynamics modeling, and ultimately to complex multimodal reasoning. Besides, in contrast to conventional per-question accuracy, we propose a \textbf{group-based non-linear evaluation} strategy that enforces both consistency across related queries and coherence in multi-step reasoning. It penalizes fragmented or guess-based correctness and assigns credit only to answers supported by valid reasoning. To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline, involving 12 annotators and 50 independent reviewers. Backed by \textbf{3,300 human-hours} and up to \textbf{5 rounds} of quality assurance, Video-MME-v2 aims to serve as one of the most authoritative video benchmarks. Extensive experiments reveal a substantial gap between current best model Gemini-3-Pro and human experts, and uncover a clear hierarchical bottleneck where errors in visual information aggregation and temporal modeling propagate to limit high-level reasoning. We further find that thinking-based reasoning is highly dependent on textual cues, improving performance with subtitles but sometimes degrading it in purely visual settings. By exposing these limitations, Video-MME-v2 establishes a demanding new testbed for the development of next-generation video MLLMs.
Abstract (translated)
URL
https://arxiv.org/abs/2604.05015
PDF
-
BoxComm: Benchmarking Category-Aware Commentary Generation and Narration Rhythm in Boxing
2026-04-06 04:56:49Kaiwen Wang, Kaili Zheng, Rongrong Deng, Yiming Shi, Chenyi Guo, Ji WuAbstract
Recent multimodal large language models (MLLMs) have shown strong capabilities in general video understanding, driving growing interest in automatic sports commentary generation. However, existing benchmarks for this task focus exclusively on team sports such as soccer and basketball, leaving combat sports entirely unexplored. Notably, combat sports present distinct challenges: critical actions unfold within milliseconds with visually subtle yet semantically decisive differences, and professional commentary contains a substantially higher proportion of tactical analysis compared to team sports. In this paper, we present BoxComm, a large-scale dataset comprising 445 World Boxing Championship match videos with over 52K commentary sentences from professional broadcasts. We propose a structured commentary taxonomy that categorizes each sentence into play-by-play, tactical, or contextual, providing the first category-level annotation for sports commentary benchmarks. Building on this taxonomy, we introduce two novel and complementary evaluations tailored to sports commentary generation: (1) category-conditioned generation, which evaluates whether models can produce accurate commentary of a specified type given video context; and (2) commentary rhythm assessment, which measures whether freely generated commentary exhibits appropriate temporal pacing and type distribution over continuous video segments, capturing a dimension of commentary competence that prior benchmarks have not addressed. Experiments on multiple state-of-the-art MLLMs reveal that current models struggle on both evaluations. We further propose EIC-Gen, an improved baseline incorporating detected punch events to supply structured action cues, yielding consistent gains and highlighting the importance of perceiving fleeting and subtle events for combat sports commentary.
Abstract (translated)
近期多模态大语言模型（MLLMs）在通用视频理解方面展现出强大能力，推动了自动体育赛事解说生成研究的兴起。然而，现有相关基准数据集仅聚焦于足球、篮球等团队运动，完全未涉及格斗类项目。值得注意的是，格斗运动存在独特挑战：关键动作在毫秒级时间内发生，视觉特征细微但语义区分度极高；且专业解说中战术分析占比显著高于团队运动。本文提出BoxComm大规模数据集，包含445场世界拳击锦标赛视频及超过5.2万条专业转播解说语句。我们设计结构化解说分类体系，将每条解说划分为现场播报、战术分析或背景说明三类，首次为体育解说基准提供细粒度标注。基于该体系，我们引入两种互补的新型评估方法：一是类别条件生成测试，评估模型能否根据视频语境生成指定类型的准确解说；二是解说节奏评估，衡量自由生成的解说在连续视频片段中是否具备恰当的时间分布与类型配比，涵盖既往基准未触及的解说能力维度。在多个先进MLLMs上的实验表明，当前模型在两个评估上均表现不佳。进一步提出EIC-Gen改进基线，通过检测出拳事件提供结构化动作线索，实现稳定性能提升，凸显感知瞬时细微事件对格斗运动解说的重要性。
URL
https://arxiv.org/abs/2604.04419
PDF
-
Structured Causal Video Reasoning via Multi-Objective Alignment
2026-04-06 04:49:30Zinuo Li, Yongxin Guo, Jun Liu, Jiawei Zhan, Xi Jiang, Chengjie Wang, Mohammed Bennamoun, Farid Boussaid, Feng Zheng, Qiuhong KeAbstract
Human understanding of video dynamics is typically grounded in a structured mental representation of entities, actions, and temporal relations, rather than relying solely on immediate deductive reasoning. In contrast, existing Video-LLMs largely depend on unstructured video reasoning, where critical visual evidence is embedded in verbose textual descriptions and temporal causality is often weakly modeled. This leads to inefficient processes and fragile causal inference. To bridge this cognitive gap, we propose constructing a compact representation of salient events and their causal relationships, which we name Structured Event Facts, prior to the reasoning stage. This structured prior serves as an explicit constraint to promote concise and causally grounded reasoning, while also making intermediate evidence easier to verify. To effectively train models on such structured facts, we introduce CausalFact-60K and a four-stage training pipeline comprising facts alignment, format warm-start, thinking warm-start, and reinforcement learning-based post-training. During RL stage, we find that this framework introduces competing objectives, as structural completeness and causal fidelity must be balanced against reasoning length, making it difficult to optimize. We address this challenge by formulating the optimization as a Multi-Objective Reinforcement Learning (MORL) problem and explicitly optimizing toward the Pareto-Frontier to balance these trade-offs. As a result, we introduce Factum-4B, which yields more reliable reasoning and delivers stronger performance on challenging video understanding tasks requiring fine-grained temporal inference.
Abstract (translated)
人类对视频动态的理解通常基于实体、动作及时序关系的结构化心智表征，而非单纯依赖即时演绎推理。相比之下，现有视频大语言模型主要依赖非结构化视频推理——关键视觉证据嵌入冗长文本描述中，时序因果关系往往建模薄弱，导致推理过程低效且因果推断脆弱。为弥合这一认知差距，我们提出在推理阶段前构建显著事件及其因果关系的紧凑表示，命名为“结构化事件事实”。该结构化先验作为显式约束，可促进简洁且具因果基础的推理，同时使中间证据更易验证。为有效训练模型掌握此类结构化事实，我们引入CausalFact-60K数据集及四阶段训练流程：事实对齐、格式预热、思维预热及基于强化学习的后训练。在强化学习阶段，我们发现该框架引入竞争性目标——结构完整性与因果保真度需与推理长度权衡，导致优化困难。我们通过将优化问题表述为多目标强化学习（MORL）并显式优化帕累托前沿来解决此挑战，以平衡这些权衡。最终推出的Factum-4B模型实现了更可靠的推理，在需要细粒度时序推断的挑战性视频理解任务上展现出更强性能。
URL
https://arxiv.org/abs/2604.04415
PDF
-
AURA: Always-On Understanding and Real-Time Assistance via Video Streams
2026-04-05 16:53:46Xudong Lu, Yang Bo, Jinpeng Chen, Shuhan Li, Xintong Guo, Huankang Guan, Fang Liu, Dunyuan Xu, Peiwen Sun, Heyang Sun, Rui Liu, Hongsheng LiAbstract
Video Large Language Models (VideoLLMs) have achieved strong performance on many video understanding tasks, but most existing systems remain offline and are not well-suited for live video streams that require continuous observation and timely response. Recent streaming VideoLLMs have made progress, yet current approaches often rely on decoupled trigger-response pipelines or are limited to captioning-style narration, reducing their effectiveness for open-ended question answering and long-horizon interaction. We propose AURA (Always-On Understanding and Real-Time Assistance), an end-to-end streaming visual interaction framework that enables a unified VideoLLM to continuously process video streams and support both real-time question answering and proactive responses. AURA integrates context management, data construction, training objectives, and deployment optimization for stable long-horizon streaming interaction. It achieves state-of-the-art performance on streaming benchmarks and supports a real-time demo system with ASR and TTS running at 2 FPS on two 80G accelerators. We release the AURA model together with a real-time inference framework to facilitate future research.
Abstract (translated)
URL
https://arxiv.org/abs/2604.04184
PDF
-
SGTA: Scene-Graph Based Multi-Modal Traffic Agent for Video Understanding
2026-04-04 12:01:49Xingcheng Zhou, Mingyu Liu, Walter Zimmer, Jiajie Zhang, Alois KnollAbstract
We present Scene-Graph Based Multi-Modal Traffic Agent (SGTA), a modular framework for traffic video understanding that combines structured scene graphs with multi-modal reasoning. It constructs a traffic scene graph from roadside videos using detection, tracking, and lane extraction, followed by tool-based reasoning over both symbolic graph queries and visual inputs. SGTA adopts ReAct to process interleaved reasoning traces from large language models with tool invocations, enabling interpretable decision-making for complex video questions. Experiments on selected TUMTraffic VideoQA dataset sample demonstrate that SGTA achieves competitive accuracy across multiple question types while providing transparent reasoning steps. These results highlight the potential of integrating structured scene representations with multi-modal agents for traffic video understanding.
Abstract (translated)
URL
https://arxiv.org/abs/2604.03697
PDF
-
KiToke: Kernel-based Interval-aware Token Compression for Video Large Language Models
2026-04-03 19:18:22Haifeng Huang, Yang LiAbstract
Video Large Language Models (Video LLMs) achieve strong performance on video understanding tasks but suffer from high inference costs due to the large number of visual tokens. We propose KiToke, a training-free, query-agnostic token compression approach that reduces spatiotemporal redundancy while preserving critical visual information. Our method estimates token diversity globally using a kernel-based redundancy measure, enabling content-adaptive selection that remains effective under extreme token budgets, and further introduces a lightweight temporal interval construction with interval-aware token merging to maintain temporal coherence. Unlike prior methods that rely on local or segment-level heuristics, KiToke explicitly captures global redundancy across an entire video, leading to more efficient token utilization. Extensive experiments on multiple video understanding benchmarks and Video LLM backbones demonstrate that KiToke consistently outperforms existing training-free compression methods, with particularly large gains at aggressive retention ratios down to 1%.
Abstract (translated)
视频大语言模型（Video LLMs）在视频理解任务上表现出色，但由于视觉标记数量庞大，导致推理成本高昂。本文提出KiToke，一种无需训练、与查询无关的标记压缩方法，可在保留关键视觉信息的同时减少时空冗余。该方法采用基于核的冗余度测量，从全局估计标记多样性，实现内容自适应选择，即使在极端标记预算下仍保持有效性，并进一步引入轻量级的时间间隔构建与间隔感知标记合并机制，以维持时间连贯性。与依赖局部或片段级启发式规则的先前方法不同，KiToke能够显式捕捉整个视频的全局冗余，从而实现更高效的标记利用。在多个视频理解基准测试及Video LLM主干网络上的大量实验表明，KiToke持续优于现有无需训练的压缩方法，尤其在低至1%的保留率下实现显著性能提升。
URL
https://arxiv.org/abs/2604.03414
PDF
-
Progressive Video Condensation with MLLM Agent for Long-form Video Understanding
2026-04-03 09:00:38Yufei Yin, Yuchen Xing, Qianke Meng, Minghao Chen, Yan Yang, Zhou YuAbstract
Understanding long videos requires extracting query-relevant information from long sequences under tight compute budgets. Existing text-then-LLM pipelines lose fine-grained visual cues, while video-based multimodal large language models (MLLMs) can keep visual details but are too frame-hungry and computationally expensive. In this work, we aim to harness MLLMs for efficient video understanding. We propose ProVCA, a progressive video condensation agent that iteratively locates key video frames at multiple granularities. ProVCA first adopts a segment localization module to identify the video segment relevant to the query, then a snippet selection module to select important snippets based on similarity, and finally a keyframe refinement module to pinpoint specific keyframes in those snippets. By progressively narrowing the scope from coarse segments to fine frames, ProVCA identifies a small set of keyframes for MLLM-based reasoning. ProVCA achieves state-of-the-art zero-shot accuracies of 69.3\% on EgoSchema, 80.5\% on NExT-QA, and 77.7\% on IntentQA, while using fewer frames than previous training-free methods.
Abstract (translated)
理解长视频需要在有限的计算预算下，从长序列中提取与查询相关的信息。现有的“文本处理+大语言模型”流程会丢失细粒度的视觉线索，而基于视频的多模态大语言模型（MLLMs）虽能保留视觉细节，却因帧数需求过高而计算成本昂贵。本研究旨在利用MLLMs实现高效视频理解。我们提出了ProVCA——一种渐进式视频压缩智能体，可迭代地在多种粒度下定位关键视频帧。ProVCA首先采用片段定位模块识别与查询相关的视频片段，接着通过片段选择模块基于相似度筛选重要片段，最终通过关键帧精炼模块在片段内确定具体关键帧。通过从粗粒度片段到细粒度帧逐步缩小范围，ProVCA为基于MLLM的推理识别出少量关键帧。在无需训练的条件下，ProVCA在EgoSchema上达到69.3%、NExT-QA上80.5%、IntentQA上77.7%的零样本准确率，且使用的帧数少于此前无训练方法。
URL
https://arxiv.org/abs/2604.02891
PDF
-
A Simple Baseline for Streaming Video Understanding
2026-04-02 17:58:04Yujiao Shen, Shulin Tian, Jingkang Yang, Ziwei LiuAbstract
Recent streaming video understanding methods increasingly rely on complex memory mechanisms to handle long video streams. We challenge this trend with a simple finding: a sliding-window baseline that feeds only the most recent N frames to an off-the-shelf VLM already matches or surpasses published streaming models. We formalize this baseline as SimpleStream and evaluate it against 13 major offline and online video LLM baselines on OVO-Bench and StreamingBench. Despite its simplicity, SimpleStream delivers consistently strong performance. With only 4 recent frames, it reaches 67.7% average accuracy on OVO-Bench and 80.59% on StreamingBench. Controlled ablations further show that the value of longer context is backbone-dependent rather than uniformly increasing with model scale, and reveal a consistent perception-memory trade-off: adding more historical context can improve recall, but often weakens real-time perception. This suggests that stronger memory, retrieval, or compression modules should not be taken as evidence of progress unless they clearly outperform SimpleStream under the same protocol. We therefore argue that future streaming benchmarks should separate recent-scene perception from long-range memory, so that performance improvements from added complexity can be evaluated more clearly.
Abstract (translated)
URL
https://arxiv.org/abs/2604.02317
PDF
-
GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
2026-04-02 14:19:58Rong Fan, Kaiyan Xiao, Minghao Zhu, Liuyi Wang, Kai Dai, Zhao YangAbstract
Video temporal grounding (VTG) is a critical task in video understanding and a key capability for extending video large language models (Vid-LLMs) to broader applications. However, existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues. To address this limitation, we propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments. GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM, thereby preserving essential spatio-temporal information and maintaining temporal coherence. Futhermore, we introduce a progressive optimization strategy that enables the LLM to effectively adapt to the non-uniform distribution of visual features, enhancing its ability to model temporal dependencies and achieve precise video localization. We comprehensively evaluate GroundVTS on three standard VTG benchmarks, where it outperforms existing methods, achieving a 7.7-point improvement in mIoU for moment retrieval and 12.0-point improvement in mAP for highlight detection. Code is available at this https URL.
Abstract (translated)
视频时间定位（VTG）是视频理解中的关键任务，也是扩展视频大语言模型（Vid-LLMs）应用范围的核心能力。然而，现有Vid-LLMs依赖均匀帧采样提取视频信息，导致关键帧分布稀疏并丢失重要时间线索。针对这一局限，我们提出了Grounded Visual Token Sampling（GroundVTS），一种聚焦于最具信息量时间片段的Vid-LLM架构。GroundVTS采用细粒度、查询引导的机制，在将视觉标记输入大语言模型前进行筛选，从而保留关键时空信息并维持时间连贯性。此外，我们引入了渐进式优化策略，使大语言模型能够有效适应视觉特征的非均匀分布，增强其建模时间依赖关系的能力，实现精准的视频定位。我们在三个标准VTG基准测试上对GroundVTS进行了全面评估，其性能超越现有方法，在时刻检索任务中mIoU提升7.7点，在亮点检测任务中mAP提升12.0点。代码已公开于此https链接。
URL
https://arxiv.org/abs/2604.02093
PDF
-
HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models
2026-04-02 10:40:15Yansong Guo, Chaoyang Zhu, Jiayi Ji, Jianghang Lin, Liujuan CaoAbstract
Video Large Language Models (VideoLLMs) have demonstrated impressive capabilities in video understanding, yet the massive number of input video tokens incurs a significant computational burden for deployment. Existing methods mainly prune video tokens at input level while neglecting the inherent information structure embedded in videos and large language models (LLMs). To address this, we propose HieraVid, a hierarchical pruning framework that progressively and dynamically reduces visual redundancy. Based on two observations that videos possess the segment-frame structure and LLMs internally propagate multi-modal information unidirectionally, we decompose pruning into three levels: 1) segment-level, where video tokens are first temporally segmented and spatially merged; 2) frame-level, where similar frames within the same segment are jointly pruned to preserve diversity; 3) layer-level, redundancy gradually shrinks as LLM layer increases w/o compromising performance. We conduct extensive experiments on four widely used video understanding benchmarks to comprehensively evaluate the effectiveness of HieraVid. Remarkably, with only 30% of tokens retained, HieraVid achieves new state-of-the-art performance, while maintaining over 98% and 99% of the performance of LLaVA-Video-7B and LLaVA-OneVision-7B, respectively.
Abstract (translated)
视频大语言模型（VideoLLMs）在视频理解方面展现出卓越能力，但海量的输入视频令牌给实际部署带来了显著的计算负担。现有方法主要在输入层面对视频令牌进行剪枝，却忽视了视频与大语言模型（LLMs）本身蕴含的信息结构。为此，我们提出HieraVid——一个分层剪枝框架，通过渐进式动态方式降低视觉冗余。基于视频具有段-帧结构、LLMs内部以单向方式传播多模态信息这两个观察，我们将剪枝分解为三个层次：1）段级，首先对视频令牌进行时间分段与空间合并；2）帧级，在同一段内联合剪除相似帧以保留多样性；3）层级，随着LLM层数增加，冗余度逐渐降低且不影响性能。我们在四个广泛使用的视频理解基准上开展大量实验，全面评估HieraVid的有效性。值得注意的是，HieraVid仅保留30%的令牌即达到新的最优性能，同时分别保持LLaVA-Video-7B与LLaVA-OneVision-7B超过98%与99%的性能表现。
URL
https://arxiv.org/abs/2604.01881
PDF
-
VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification
2026-04-02 03:29:43Jiahao Meng, Tan Yue, Qi Xu, Haochen Wang, Zhongwei Ren, Weisong Liu, Yuhao Wang, Renrui Zhang, Yunhai Tong, Haodong DuanAbstract
Recent video multimodal large language models achieve impressive results across various benchmarks. However, current evaluations suffer from two critical limitations: (1) inflated scores can mask deficiencies in fine-grained visual understanding and reasoning, and (2) answer correctness is often measured without verifying whether models identify the precise spatio-temporal evidence supporting their predictions. To address this, we present VideoZeroBench, a hierarchical benchmark designed for challenging long-video question answering that rigorously verifies spatio-temporal evidence. It comprises 500 manually annotated questions across 13 domains, paired with temporal intervals and spatial bounding boxes as evidence. To disentangle answering generation, temporal grounding, and spatial grounding, we introduce a five-level evaluation protocol that progressively tightens evidence requirements. Experiments show that even Gemini-3-Pro correctly answers fewer than 17% of questions under the standard end-to-end QA setting (Level-3). When grounding constraints are imposed, performance drops sharply: No model exceeds 1% accuracy when both correct answering and accurate spatio-temporal localization are required (Level-5), with most failing to achieve any correct grounded predictions. These results expose a significant gap between surface-level answer correctness and genuine evidence-based reasoning, revealing that grounded video understanding remains a bottleneck for long-video QA. We further analyze performance across minimal evidence spans, atomic abilities, and inference paradigms, providing insights for future research in grounded video reasoning. The benchmark and code will be made publicly available.
Abstract (translated)
近期视频多模态大语言模型在各种基准测试中取得了令人瞩目的成果。然而，当前评估存在两个关键局限：(1) 虚高的分数可能掩盖模型在细粒度视觉理解与推理方面的缺陷；(2) 答案正确性的判定往往未验证模型是否识别出支撑其预测的精确时空证据。为解决此问题，我们提出了 VideoZeroBench——一个面向挑战性长视频问答的层级化基准，可严格验证时空证据。该基准包含13个领域中500个手工标注的问题，并配以时间区间和空间边界框作为证据。为解耦答案生成、时序定位与空间定位，我们设计了五级评估协议，逐步收紧证据要求。实验表明，即使在标准端到端问答设置（Level-3）下，Gemini-3-Pro 正确回答的问题比例仍不足17%。当施加定位约束时，性能急剧下降：在同时要求答案正确与时空定位准确（Level-5）的条件下，所有模型准确率均未超过1%，多数模型甚至无法产生任何正确的定位预测。这些结果揭示了表面答案正确性与真实基于证据的推理之间存在显著差距，表明基于定位的视频理解仍是长视频问答的瓶颈。我们还针对最小证据跨度、原子能力及推理范式进行了性能分析，为未来基于定位的视频推理研究提供见解。该基准及代码将公开发布。
URL
https://arxiv.org/abs/2604.01569
PDF
-
Reinforcing Consistency in Video MLLMs with Structured Rewards
2026-04-01 23:15:34Yihao Quan, Zeru Shi, Jinman Zhao, Ruixiang TangAbstract
Multimodal large language models (MLLMs) have achieved remarkable progress in video understanding. However, seemingly plausible outputs often suffer from poor visual and temporal grounding: a model may fabricate object existence, assign incorrect attributes, or collapse repeated events while still producing a globally reasonable caption or answer. We study this failure mode through a compositional consistency audit that decomposes a caption into supporting factual and temporal claims, investigating whether a correct high-level prediction is actually backed by valid lower-level evidence. Our top-down audit reveals that even correct root relational claims often lack reliable attribute and existence support. This indicates that standard sentence-level supervision is a weak proxy for faithful video understanding. Furthermore, when turning to reinforcement learning (RL) for better alignment, standard sentence-level rewards often prove too coarse to accurately localize specific grounding failures. To address this, we replace generic sentence-level rewards with a structured reward built from factual and temporal units. Our training objective integrates three complementary components: (1) an instance-aware scene-graph reward for factual objects, attributes, and relations; (2) a temporal reward for event ordering and repetition; and (3) a video-grounded VQA reward for hierarchical self-verification. Across temporal, general video understanding, and hallucination-oriented benchmarks, this objective yields consistent gains on open-source backbones. These results suggest that structured reward shaping is a practical route to more faithful video understanding.
Abstract (translated)
多模态大语言模型（MLLMs）在视频理解领域已取得显著进展。然而，看似合理的输出往往存在视觉与时间定位不准确的问题：模型可能虚构物体存在、赋予错误属性，或在重复事件处理上出现偏差，却仍生成整体通顺的说明或回答。本研究通过组合一致性审计方法，将说明分解为支撑性事实主张与时间主张，探究高层预测是否真正具备有效的底层证据支持。这种自上而下的审计揭示，即使正确的关系性主张也常缺乏可靠的属性与存在性支撑，表明标准句子级监督对忠实视频理解而言是较弱的代理指标。此外，当转向强化学习（RL）以优化对齐时，标准句子级奖励往往过于粗糙，难以精确定位具体定位失败。为此，我们采用基于事实与时间单元的结构化奖励替代通用句子级奖励。训练目标整合三个互补组件：（1）面向实例的场景图奖励，用于事实性对象、属性与关系；（2）针对事件排序与重复的时间奖励；（3）用于分层自验证的视频基础VQA奖励。在时间理解、通用视频理解及面向幻觉的基准测试中，该目标在开源模型上均实现稳定提升。结果表明，结构化奖励塑造是提升视频理解忠实度的可行路径。
URL
https://arxiv.org/abs/2604.01460
PDF
-
Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding
2026-04-01 15:02:50Yiheng Wang, Lichen Zhu, Yueqian Lin, Yudong Liu, Jingyang Zhang, Hai "Helen" Li, Yiran ChenAbstract
Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering, but their application to long-form videos is constrained by limited context length and computational cost, making keyframe sampling essential. Existing approaches typically rely on semantic relevance or reinforcement learning, which either fail to capture evidential clues or suffer from inefficient combinatorial optimization. In this work, we propose an evidence-driven keyframe sampling framework grounded in information bottleneck theory. We formulate keyframe selection as maximizing the conditional mutual information between selected frames and the query, providing a principled objective that reflects each frame's contribution to answering the question. To make this objective tractable, we exploit its structure to derive a decomposed optimization that reduces subset selection to independent frame-level scoring. We further introduce a query-conditioned evidence scoring network trained with a contrastive objective to estimate evidential importance efficiently. Experiments on long-form video understanding benchmarks show that our method consistently outperforms prior sampling strategies under strict token budgets, while significantly improving training efficiency.
Abstract (translated)
多模态大语言模型（MLLMs）在视频问答任务中展现出卓越性能，但其应用于长视频时受限于上下文长度与计算成本，使得关键帧采样成为必要环节。现有方法通常依赖语义相关性或强化学习，前者难以捕捉证据性线索，后者则面临组合优化效率低下的问题。本研究提出一个基于信息瓶颈理论的证据驱动关键帧采样框架，将关键帧选择建模为最大化所选帧与查询之间的条件互信息，从而建立能反映各帧对回答问题贡献的原则性目标。为实现该目标的可计算性，我们利用其结构特性推导出分解优化方案，将子集选择问题转化为独立的帧级评分。进一步引入经对比学习目标训练的查询条件证据评分网络，以高效估计帧的证据重要性。在长视频理解基准测试中，本方法在严格令牌预算下持续优于现有采样策略，同时显著提升训练效率。
URL
https://arxiv.org/abs/2604.01002
PDF
-
Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment
2026-04-01 13:55:28Zhuchenyang Liu, Yao Zhang, Yu XiaoAbstract
2D assembly diagrams are often abstract and hard to follow, creating a need for intelligent assistants that can monitor progress, detect errors, and provide step-by-step guidance. In mixed reality settings, such systems must recognize completed and ongoing steps from the camera feed and align them with the diagram instructions. Vision Language Models (VLMs) show promise for this task, but face a depiction gap because assembly diagrams and video frames share few visual features. To systematically assess this gap, we construct IKEA-Bench, a benchmark of 1,623 questions across 6 task types on 29 IKEA furniture products, and evaluate 19 VLMs (2B-38B) under three alignment strategies. Our key findings: (1) assembly instruction understanding is recoverable via text, but text simultaneously degrades diagram-to-video alignment; (2) architecture family predicts alignment accuracy more strongly than parameter count; (3) video understanding remains a hard bottleneck unaffected by strategy. A three-level mechanistic analysis further reveals that diagrams and video occupy disjoint ViT subspaces, and that adding text shifts models from visual to text-driven reasoning. These results identify visual encoding as the primary target for improving cross-depiction robustness. Project page: this https URL
Abstract (translated)
二维装配图通常抽象且难以理解，这催生了对能够监控进度、检测错误并提供逐步指导的智能助手的需求。在混合现实场景中，此类系统必须通过摄像头画面识别已完成和进行中的步骤，并将其与图示指令对齐。视觉语言模型（VLMs）在此任务中展现出潜力，但面临表征鸿沟——装配图与视频帧共享的视觉特征极少。为系统评估这一差距，我们构建了IKEA-Bench基准测试，涵盖29款宜家家具产品的6类任务共1623个问题，并评估了19个（20亿至380亿参数）视觉语言模型在三种对齐策略下的表现。核心发现如下：(1) 装配指令理解可通过文本恢复，但文本同时削弱了图-视频对齐能力；(2) 架构家族比参数量更显著影响对齐精度；(3) 视频理解仍是策略无法缓解的关键瓶颈。三级机理分析进一步揭示：图示与视频占据不同的ViT子空间，且添加文本会使模型从视觉推理转向文本驱动推理。这些结果指出，视觉编码是提升跨表征鲁棒性的主要优化方向。项目页面：this https URL
URL
https://arxiv.org/abs/2604.00913
PDF
-
An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models
2026-04-01 11:45:28Lennart Maack, Alexander SchlaeferAbstract
Surgical video understanding is a crucial prerequisite for advancing Computer-Assisted Surgery. While vision-language models (VLMs) have recently been applied to the surgical domain, existing surgical vision-language datasets lack in capturing and evaluating complex, interleaved spatial-temporal dynamics. Creating large scale datasets that accurately represent fine-grained spatial-temporal relationships in surgical videos is challenging due to costly manual annotations or error-prone generation using large language models. To address this gap, we introduce the SurgSTU-Pipeline, a deterministic generation pipeline featuring temporal and spatial continuity filtering to reliably create surgical datasets for fine-grained spatial-temporal multimodal understanding. Applying this pipeline to publicly available surgical datasets, we create the SurgSTU dataset, comprising 7515 video clips densely extended with 150k fine-grained spatial-temporal question-answer samples. Our comprehensive evaluation shows that while state-of-the-art generalist VLMs struggle in zero-shot settings, their spatial-temporal capabilities can be improved through in-context learning. A fine-tuned VLM on the SurgSTU training dataset achieves highest performance among all spatial-temporal tasks, validating the dataset's efficacy to improve spatial-temporal understanding of VLMs in surgical videos. Code will be made publicly available.
Abstract (translated)
手术视频理解是推进计算机辅助手术的关键前提。尽管视觉语言模型（VLMs）近期已被应用于外科领域，但现有手术视觉语言数据集在捕捉和评估复杂交织的时空动态方面存在不足。由于人工标注成本高昂，或使用大语言模型生成时易产生错误，创建能准确表征手术视频中细粒度时空关系的大规模数据集极具挑战。为填补此空白，我们提出了SurgSTU-Pipeline——一种具备时空连续性过滤功能的确定性生成流程，可可靠地创建用于细粒度时空多模态理解的手术数据集。通过将该流程应用于公开手术数据集，我们构建了SurgSTU数据集，该数据集包含7515个视频片段，并密集扩展了15万条细粒度时空问答样本。全面评估表明，尽管最先进的通用视觉语言模型在零样本设置下表现欠佳，但其时空能力可通过上下文学习得到提升。在SurgSTU训练数据集上微调的视觉语言模型在所有时空任务中达到最高性能，验证了该数据集对提升视觉语言模型手术视频时空理解能力的有效性。代码将公开提供。
URL
https://arxiv.org/abs/2604.00784
PDF
-
IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models
2026-04-01 11:23:16Dong-Jae Lee, Sunghyun Baek, Junmo KimAbstract
Large Vision Language Models show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens. Existing token pruning methods mitigate this issue through empirical approaches while overlooking the internal mechanism of attention. In this paper, we propose a novel training free token pruning framework grounded in the dual form perspective of attention. We reformulate attention as an implicit linear layer whose weight matrix is the sum of rank 1 outer products, each generated by a single token's key value pair. Token pruning thus reduces to selecting an optimal subset of these rank 1 updates that best approximates the original dual weight matrix. Extending this perspective to standard softmax attention in LVLMs, we derive a novel metric quantifying both a token's information magnitude and information duplication. To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance. Extensive experiments demonstrate that our method achieves a better trade off between performance and efficiency, while providing another perspective on existing pruning approaches.
Abstract (translated)
大型视觉语言模型在图像与视频理解任务中展现出卓越性能，但其计算成本随视觉token数量急剧增加。现有token剪枝方法多依赖经验性策略，却忽视了注意力的内在机制。本文提出一种基于注意力对偶形式视角的新型无训练token剪枝框架。我们将注意力重构为一个隐式线性层，其权重矩阵由多个秩1外积之和构成，每个外积由单个token的键值对生成。因此，token剪枝问题转化为选择最优子集，使这些秩1更新能最佳逼近原始对偶权重矩阵。将该视角扩展至LVLM的标准softmax注意力，我们提出一种同时量化token信息量及信息冗余的新指标。为高效筛选指标定义的子集，我们设计了渐进分块最大边际相关性算法。大量实验表明，该方法在性能与效率间取得更优平衡，并为现有剪枝方法提供了新的理论视角。
URL
https://arxiv.org/abs/2604.00757
PDF
