Visual_Relation
-
Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
2026-07-07 01:00:51Zhiwei Yang, Yuanchen Wu, Nan Zhang, Yucong Meng, Ke Yan, Shouhong DingAbstract
Multimodal Large Language Models (MLLMs) have demonstrated strong perception and reasoning capabilities. However, most existing models focus on isolated objects and neglect structured relationships for efficient target navigation, limiting their performance on visually intensive tasks. To address this challenge, we introduce Scene Graph Thinking (SaGe), a novel paradigm that enables fine-grained and structured visual reasoning through explicit scene-graph representations. Specifically, we first introduce an automated data engine that converts flat image-text corpora into structured scene graphs, where hierarchical entities constitute the nodes and diverse visual relations define the edges. Building upon this, we construct 120K high-quality training data by sampling reasoning traces from scene graphs. Then, two-stage graph-aligned post-training paradigms are introduced, where supervised fine-tuning internalizes MLLMs with structured reasoning, and subsequent reinforcement fine-tuning proposes node-as-proxy graph rewards to consolidate efficient graph exploration. With curated data and graph-aligned training, our approach achieves significant improvements across eight multimodal benchmarks, demonstrating strong effectiveness on fine-grained perception and reasoning tasks. Code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2607.05716
PDF
-
MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein
2026-06-28 15:39:05Hong-Han Wang, Yuntao Wang, Hu DingAbstract
Multimodal Large Language Models (MLLMs) inherit rich relational priors from their language backbones, yet often fail when asked to apply these relationships in visual contexts. We trace this failure to a structural blind spot: projection-based alignment trains each visual token to carry the right semantics, but never asks whether the relationships between concepts survive the crossing from language to vision. To address this, we propose MIRROR (Mapping Inter-concept Relations from language to visual Representation via Optimal-transport-based Regularization), a geometric regularization framework that transfers relational priors from language to vision by exploiting the rich relational structure encoded in language representations. Specifically, we derive a surrogate loss from the proposed Semi-Inverse Gromov-Wasserstein (SI-GW) problem, an inverse geometric problem that aligns visual representations with language-derived relational priors. We show that this formulation admits a unique closed-form solution that prescribes the ideal visual relational structure implied by language geometry and cross-modal coupling. The structure of the formulation also enables efficient computation, making it applicable to long token sequences. Applying SI-GW inside decoder-only Transformers requires careful design. We introduce targeted strategies at the layer, head, and token levels to ensure stable extraction without additional parameters or inference cost. MIRROR improves relational consistency while preserving performance on general vision-language tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2606.29462
PDF
-
HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoning
2026-06-04 12:40:15Moshiur Farazi, Sameera Ramasinghe, Mahbub Ahmed Turza, Shafin RahmanAbstract
Vision-Language Models (VLMs) struggle with compositional reasoning that requires understanding inter-object relationships. A natural remedy is to inject explicit scene graph triplets $\langle s, p, o \rangle$ from an off-the-shelf scene graph generator (SGG), but we show this backfires: discrete text labels collide with the continuous visual modality, degrading GQA accuracy from 60.38\% to 58.86\%. We propose \textbf{HyperVis}, which bypasses the SGG semantic bottleneck entirely. From $N$ class-agnostic region proposals, we compute a dense $O(N^2)$ visual relation tensor via spatially-biased cross-attention, project it onto a Lorentz hyperboloid, and enforce hierarchy through spatial physics, namely IoA-driven entailment cones and exterior-angle repulsion. We discover that HyperVis contributes in two complementary ways: (1) as a \emph{training-time regularizer}, the hyperbolic relational losses shape LoRA representations that improve generative VQA (GQA 61.03\% vs.\ 57.21\% for LoRA fine-tuning without relational losses, recovering and surpassing the baseline); and (2) as an \emph{inference-time relational encoder}, hyperbolic prefix tokens boost discriminative compositional scoring (SugarCrepe 79.94\%, $+$6.25pp over baseline). The learned curvature stabilises at $\kappa{=}4.0$, an order of magnitude above prior hyperbolic VLMs where $\kappa$ typically collapses toward zero, indicating that continuous visual features genuinely require the exponential volume of strongly curved space. A controlled Euclidean ablation confirms this decomposition: the relational pipeline regularises LoRA comparably in flat space (GQA 60.81\%), but the compositionality gain is specifically hyperbolic (SugarCrepe $+$4.58pp over Euclidean), with entailment loss ${\sim}6{\times}$ higher in Euclidean training. Codes are available at TBA.
Abstract (translated)
URL
https://arxiv.org/abs/2606.06100
PDF
-
Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphs
2026-06-04 09:23:08Yi Chen, Yinghao Lu, Zhehao Li, Chenchen Yan, Jiafei Wu, Chong Wang, Jiangbo QianAbstract
Open-vocabulary object detection seeks to identify novel object categories that were not part of the training data. Many knowledge distillation-based approaches have shown promising performance by transferring knowledge from pre-trained vision-language models to object detection. However, these methods often overlook structured, image-specific relationships between objects, such as interactions and spatial arrangements. This oversight can significantly restrict the effectiveness of detecting novel categories. To address this issue, we propose a Scene-guided Relational Modeling detection framework. This framework utilizes scene graphs to capture structured semantic and spatial relationships between candidate regions and their contextual objects. It explicitly models interactions among neighboring regions and incorporates a Relation Attention Module to implicitly amplify the key relational cues extracted from the scene graph. Furthermore, we present a scene-based textual alignment branch that distills category knowledge from captions to guide relational alignment. This approach facilitates a seamless integration of visual relations with semantic information for enhanced detection performance. Comprehensive experiments show that our model achieves superior performance compared to other OVOD methods, improving the AP for novel categories on COCO and LVIS datasets.
Abstract (translated)
URL
https://arxiv.org/abs/2606.05916
PDF
-
Answer Self-Consistency with Margin-Triggered Question Re-Arbitration for the CVPR 2026 VidLLMs Challenge
2026-06-03 00:51:55Tomoya Miyazawa, Hiroyasu OkunoAbstract
In this report, we present our solution for Track 2 of the CVPR 2026 VidLLMs Challenge. This track evaluates visual relational reasoning in videos, where models must infer relations that are not always explicitly visible. We propose Answer Self-Consistency with Margin-Triggered Question Re-Arbitration (ASC-MQRA), a training-free test-time reasoning framework built on a multimodal reasoning model. The core ASC component performs multiple stochastic video question-answering runs and aggregates their answer choices through answer-level self-consistency. This substantially improves over single-pass inference and forms our final test submission. We further study MQRA, a conditional re-arbitration module for low-margin examples where the first-stage vote distribution indicates uncertainty. Our vote-margin analysis shows that low-margin examples often retain the ground-truth answer among the top candidates, motivating MQRA to narrow the candidate set and re-watch the video only over the retained candidates. On validation, MQRA further improves over ASC, indicating that low-margin vote distributions can provide a useful uncertainty signal. On test, however, MQRA slightly degrades performance relative to ASC, suggesting that re-arbitration is sensitive to the size and category distribution of the triggered subset. Our final test submission therefore uses ASC without re-arbitration, achieving 72.73 average accuracy and 78.34 category-wise macro average accuracy on validation, and 81.16 average accuracy and 80.91 category-wise macro average accuracy on test. This report details our prompting strategy, implementation setup, ablation studies, and diagnostic analyses. The code is available at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2606.04323
PDF
-
Question-Aware Evidence Ledgers for Video Relational Reasoning
2026-06-01 17:18:03Yilin Ou, Mengshi Qi, Huadong MaAbstract
The VRR-QA challenge evaluates visual relational reasoning in videos, where answers often depend on implicit spatial relations, event boundaries, target identity, and dialogue context rather than a single salient frame. We present a test-time reasoning pipeline built around a strong GPT-5.5 video QA solver and a set of question-aware evidence ledgers. The initial solver answers each question from a uniform video representation, while routed ledgers are prompted to make the required targets, count units, reference frames, and temporal or spatial scope explicit for counting, spatial, endpoint, viewpoint, and dialogue reasoning. External tools such as open-vocabulary detection, depth cues, pair crops, ASR, and scene-graph ledgers are used only as evidence sources. A conservative gate keeps the current answer unless independent evidence uniquely supports a different option. The final evidence-gated pipeline achieves 92.95% overall accuracy and 93.79% macro accuracy on the challenge test split.
Abstract (translated)
URL
https://arxiv.org/abs/2606.02506
PDF
-
VisAnalog: A Diagnostic Suite for Visual Concept Transfer on Natural Images
2026-05-22 01:43:32Zhaonan Li, Kyle R. Chickering, Bangzheng Li, Jacob Dineen, Xiao Ye, Zhikun Xu, Shijie Lu, Yuxi Huang, Ming Shen, Bach Nguyen, Jaya Adithya Pavuluri, Mau Son Nguyen, Sanika Chavan, Ngoc Minh Thu Le, Muhao Chen, Ben ZhouAbstract
A useful test of visual concept learning is not just whether a model can recognize a concept in a single image, but whether it can preserve and manipulate concept-level properties under transformation and transfer them to new scenes. We introduce VisAnalog, a controlled suite for this setting on natural images. Each example instantiates $A\!:\!B::C\!:\,?$: images $B$ and a hidden target image $D$ are produced by applying the same deterministic transformation sequence to source images $A$ and $C$. Given $A$, $B$, and $C$, a model must answer a multiple-choice question about $D$. The benchmark contains 617 human-validated questions spanning one- to four-step transformations such as zoom, quadrant swap, rotation, flip, and hue rotation. Across strong proprietary and open-source VLMs, end-to-end accuracy is substantially lower than oracle accuracy when $D$ is directly shown, and degrades sharply as transformation depth increases, while human performance remains near the ceiling. A program-conditioned evaluation further separates failures of relation inference from failures of transformation application, showing that inferring the visual relation from $A \rightarrow B$ is the dominant bottleneck, with additional application errors emerging on harder multi-step cases. The dataset is publicly available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2605.23141
PDF
-
MOSA: Motion-Guided Semantic Alignment for Dynamic Scene Graph Generation
2026-04-21 16:18:26Xuejiao Wang, Bohao Zhang, Changbo Wang, Gaoqi HeAbstract
Dynamic Scene Graph Generation (DSGG) aims to structurally model objects and their dynamic interactions in video sequences for high-level semantic understanding. However, existing methods struggle with fine-grained relationship modeling, semantic representation utilization, and the ability to model tail relationships. To address these issues, this paper proposes a motion-guided semantic alignment method for DSGG (MoSA). First, a Motion Feature Extractor (MFE) encodes object-pair motion attributes such as distance, velocity, motion persistence, and directional consistency. Then, these motion attributes are fused with spatial relationship features through the Motion-guided Interaction Module (MIM) to generate motion-aware relationship representations. To further enhance semantic discrimination capabilities, the cross-modal Action Semantic Matching (ASM) mechanism aligns visual relationship features with text embeddings of relationship categories. Finally, a category-weighted loss strategy is introduced to emphasize learning of tail relationships. Extensive and rigorous testing shows that MoSA performs optimally on the Action Genome dataset.
Abstract (translated)
URL
https://arxiv.org/abs/2604.19631
PDF
-
Can We Build Scene Graphs, Not Classify Them? FlowSG: Progressive Image-Conditioned Scene Graph Generation with Flow Matching
2026-04-18 01:48:43Xin Hu, Ke Qin, Wen Yin, Yuan-Fang Li, Ming Li, Tao HeAbstract
Scene Graph Generation (SGG) unifies object localization and visual relationship reasoning by predicting boxes and subject-predicate-object triples. Yet most pipelines treat SGG as a one-shot, deterministic classification problem rather than a genuinely progressive, generative task. We propose FlowSG, which recasts SGG as continuous-time transport on a hybrid discrete-continuous state: starting from a noised graph, the model progressively grows an image-conditioned scene graph through constraint-aware refinements that jointly synthesize nodes (objects) and edges (predicates). Specifically, we first leverage a VQ-VAE to quantize a scene graph (e.g., continuous visual features) into compact, predictable tokens; a graph Transformer then (i) predicts a conditional velocity field to transport continuous geometry (boxes) and (ii) updates discrete posteriors for categorical tokens (object features and predicate labels), coupling semantics and geometry via flow-conditioned message aggregation. Training combines flow-matching losses for geometry with a discrete-flow objective for tokens, yielding few-step inference and plug-and-play compatibility with standard detectors and segmenters. Extensive experiments on VG and PSG under closed- and open-vocabulary protocols show consistent gains in predicate R/mR and graph-level metrics, validating the mixed discrete-continuous generative formulation over one-shot classification baselines, with an average improvement of about 3 points over the state-of-the-art USG-Par.
Abstract (translated)
URL
https://arxiv.org/abs/2604.18623
PDF
-
From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception
2026-04-14 09:32:13Jilong Zhu, Yang FengAbstract
While Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in general visual understanding, they frequently falter in fine-grained perception tasks that require identifying tiny objects or discerning subtle visual relationships. We attribute this limitation to Visual Attenuation: a phenomenon where sparse fine-grained visual signals are prematurely suppressed or diluted by dominant textual tokens during network propagation, resulting in a "loss of focus" during the deep-level decision-making process. Existing input-centric solutions fail to fundamentally reverse this intrinsic mechanism of information loss. To address this challenge, we propose the Variational Information Flow (VIF) framework. Adopting a probabilistic perspective, VIF leverages a Conditional Variational Autoencoder (CVAE) to model the visual saliency relevant to the question-answer pair as a latent distribution. As a plug-and-play module, VIF can be integrated into existing architectures. Extensive evaluations across diverse benchmarks, covering General VQA, fine-grained perception, and visual grounding, demonstrate that VIF yields competitive improvements over previous methods, validating its effectiveness in enhancing the fine-grained perception of MLLMs.
Abstract (translated)
URL
https://arxiv.org/abs/2604.12508
PDF
-
Chart-RL: Policy Optimization Reinforcement Learning for Enhanced Visual Reasoning in Chart Question Answering with Vision Language Models
2026-04-03 16:28:03Yunfei Bai, Amit Dhanda, Shekhar JainAbstract
The recent advancements in Vision Language Models (VLMs) have demonstrated progress toward true intelligence requiring robust reasoning capabilities. Beyond pattern recognition, linguistic reasoning must integrate with visual comprehension, particularly for Chart Question Answering (CQA) tasks involving complex data visualizations. Current VLMs face significant limitations in CQA, including imprecise numerical extraction, difficulty interpreting implicit visual relationships, and inadequate attention mechanisms for capturing spatial relationships in charts. In this work, we address these challenges by presenting Chart-RL, a novel reinforcement learning framework that enhances VLMs chart understanding through feedback-driven policy optimization of visual perception and logical inference. Our key innovation includes a comprehensive framework integrating Reinforcement Learning (RL) from Policy Optimization techniques along with adaptive reward functions, that demonstrates superior performance compared to baseline foundation models and competitive results against larger state-of-the-art architectures. We also integrated Parameter-Efficient Fine-Tuning through Low-Rank Adaptation (LoRA) in the RL framework that only requires single GPU configurations while preserving performance integrity. We conducted extensive benchmarking across open-source, proprietary, and state-of-the-art closed-source models utilizing the ChartQAPro dataset. The RL fine-tuned Qwen3-VL-4B-Instruct model achieved an answer accuracy of 0.634, surpassing the 0.580 accuracy of the Qwen3-VL-8B-Instruct foundation model despite utilizing half the parameter count, while simultaneously reducing inference latency from 31 seconds to 9 seconds.
Abstract (translated)
视觉语言模型（VLMs）的最新进展已展现出向真正智能迈进的态势，而这种智能需要强大的推理能力。超越模式识别，语言推理必须与视觉理解相结合，尤其在涉及复杂数据可视化的图表问答（CQA）任务中。当前VLMs在CQA方面面临显著局限，包括数值提取不精确、难以解读隐含视觉关系，以及对图表中空间关系的注意力机制不足。本研究通过提出Chart-RL——一种新颖的强化学习框架，来应对这些挑战。该框架通过视觉感知与逻辑推理的反馈驱动策略优化，增强了VLMs的图表理解能力。我们的核心创新在于整合了基于策略优化的强化学习（RL）技术与自适应奖励函数的综合框架，相较于基线基础模型展现出更优性能，并与大型先进架构相比也取得了具有竞争力的结果。我们还通过低秩适配（LoRA）在RL框架中集成了参数高效微调，仅需单GPU配置即可保持性能完整性。我们利用ChartQAPro数据集，对开源、专有及最先进的闭源模型进行了广泛基准测试。经RL微调的Qwen3-VL-4B-Instruct模型达到了0.634的答案准确率，超越了参数量减半的Qwen3-VL-8B-Instruct基础模型（0.580准确率），同时将推理延迟从31秒降至9秒。
URL
https://arxiv.org/abs/2604.03157
PDF
-
REACT++: Efficient Cross-Attention for Real-Time Scene Graph Generation
2026-03-06 15:36:00Ma\"elic Neau, Zoe FalomirAbstract
Scene Graph Generation (SGG) is a task that encodes visual relationships between objects in images as graph structures. SGG shows significant promise as a foundational component for downstream tasks, such as reasoning for embodied agents. To enable real-time applications, SGG must address the trade-off between performance and inference speed. However, current methods tend to focus on one of the following: (1) improving relation prediction accuracy, (2) enhancing object detection accuracy, or (3) reducing latency, without aiming to balance all three objectives simultaneously. To address this limitation, we build on the powerful Real-time Efficiency and Accuracy Compromise for Tradeoffs in Scene Graph Generation (REACT) architecture and propose REACT++, a new state-of-the-art model for real-time SGG. By leveraging efficient feature extraction and subject-to-object cross-attention within the prototype space, REACT++ balances latency and representational power. REACT++ achieves the highest inference speed among existing SGG models, improving relation prediction accuracy without sacrificing object detection performance. Compared to the previous REACT version, REACT++ is 20% faster with a gain of 10% in relation prediction accuracy on average. The code is available at this https URL.
Abstract (translated)
场景图生成（SGG）是一项任务，旨在将图像中对象之间的视觉关系编码为图结构。SGG作为下游任务的基础组件表现出巨大潜力，例如用于具身代理的推理。为了实现实时应用，SGG必须解决性能和推断速度之间的权衡问题。然而，目前的方法往往集中于以下三个目标之一：(1) 提高关系预测准确性；(2) 增强对象检测准确性；或 (3) 减少延迟，并没有同时兼顾这三个目标。 为了解决这一局限性，我们基于强大的“实时效率与精度权衡的场景图生成架构”（REACT）构建了 REACT++，这是一种新的、面向实时SGG的状态-of-the-art模型。通过利用高效的特征提取和原型空间内的主客体交叉注意机制，REACT++ 平衡了延迟和表示能力之间的关系。REACT++ 达到了现有 SGG 模型中最快的推断速度，在不牺牲对象检测性能的情况下提高了关系预测准确性。 与之前的 REACT 版本相比，REACT++ 在关系预测准确性的平均值上提升了10%，并且速度快20%。代码可在该网址获得：[此链接指向的是原文中的URL位置，请根据实际情况提供或省略]。
URL
https://arxiv.org/abs/2603.06386
PDF
-
Predictive Reasoning with Augmented Anomaly Contrastive Learning for Compositional Visual Relations
2026-03-01 14:17:44Chengtai Li, Yuting He, Jianfeng Ren, Ruibin Bai, Yitian Zhao, Heng Yu, Xudong JiangAbstract
While visual reasoning for simple analogies has received significant attention, compositional visual relations (CVR) remain relatively unexplored due to their greater complexity. To solve CVR tasks, we propose Predictive Reasoning with Augmented Anomaly Contrastive Learning (PR-A$^2$CL), \ie, to identify an outlier image given three other images that follow the same compositional rules. To address the challenge of modelling abundant compositional rules, an Augmented Anomaly Contrastive Learning is designed to distil discriminative and generalizable features by maximizing similarity among normal instances while minimizing similarity between normal and anomalous outliers. More importantly, a predict-and-verify paradigm is introduced for rule-based reasoning, in which a series of Predictive Anomaly Reasoning Blocks (PARBs) iteratively leverage features from three out of the four images to predict those of the remaining one. Throughout the subsequent verification stage, the PARBs progressively pinpoint the specific discrepancies attributable to the underlying rules. Experimental results on SVRT, CVR and MC$^2$R datasets show that PR-A$^2$CL significantly outperforms state-of-the-art reasoning models.
Abstract (translated)
尽管简单的类比视觉推理已经得到了广泛关注，但组合视觉关系（CVR）由于其更大的复杂性而相对较少被探索。为了解决CVR任务，我们提出了预测增强异常对比学习的预测推理(PR-A²CL)，即在给定三个遵循相同组合规则的图像的情况下识别一个离群图像。为了应对建模丰富组合规则的挑战，设计了增强异常对比学习以通过最大化正常实例之间的相似性同时最小化正常实例与异常离群之间的相似性来提取区分性和泛化的特征。更重要的是，引入了一种基于规则推理的预测和验证范式，在此过程中一系列预测异常推理模块(PARBs)迭代地利用四个图像中的三个图像的特征来预测剩余一个图像的特征。在随后的验证阶段，PARBs逐步确定归因于底层规则的具体差异。 实验结果表明，在SVRT、CVR 和 MC²R 数据集上，PR-A²CL 显著优于最先进的推理模型。
URL
https://arxiv.org/abs/2603.01125
PDF
-
Graph Recognition via Subgraph Prediction
2026-01-21 16:07:17Andr\'e Eberhard, Gerhard Neumann, Pascal FriederichAbstract
Despite tremendous improvements in tasks such as image classification, object detection, and segmentation, the recognition of visual relationships, commonly modeled as the extraction of a graph from an image, remains a challenging task. We believe that this mainly stems from the fact that there is no canonical way to approach the visual graph recognition task. Most existing solutions are specific to a problem and cannot be transferred between different contexts out-of-the box, even though the conceptual problem remains the same. With broad applicability and simplicity in mind, in this paper we develop a method, \textbf{Gra}ph Recognition via \textbf{S}ubgraph \textbf{P}rediction (\textbf{GraSP}), for recognizing graphs in images. We show across several synthetic benchmarks and one real-world application that our method works with a set of diverse types of graphs and their drawings, and can be transferred between tasks without task-specific modifications, paving the way to a more unified framework for visual graph recognition.
Abstract (translated)
尽管在图像分类、目标检测和分割等任务上取得了巨大的进步，但识别视觉关系（通常建模为从图像中提取图形）仍然是一项具有挑战性的任务。我们认为这主要是由于目前没有一个标准的方式来处理视觉图识别任务。大多数现有的解决方案都是针对特定问题的，无法直接应用于不同的情境中，尽管概念性的问题是一样的。本着广泛适用性和简单性的原则，在本文中我们开发了一种方法——通过子图预测进行图形识别（GraSP），用于在图像中识别图形。我们在多个合成基准测试和一个现实世界的应用程序上展示了该方法可以处理多种不同类型的图及其绘制方式，并且可以在任务之间转移，而无需特定任务的修改，这为视觉图识别的一个更加统一框架铺平了道路。
URL
https://arxiv.org/abs/2601.15133
PDF
-
SliceLens: Fine-Grained and Grounded Error Slice Discovery for Multi-Instance Vision Tasks
2025-12-31 03:28:41Wei Zhang, Chaoqun Wang, Zixuan Guan, Sam Kao, Pengfei Zhao, Peng Wu, Sifeng HeAbstract
Systematic failures of computer vision models on subsets with coherent visual patterns, known as error slices, pose a critical challenge for robust model evaluation. Existing slice discovery methods are primarily developed for image classification, limiting their applicability to multi-instance tasks such as detection, segmentation, and pose estimation. In real-world scenarios, error slices often arise from corner cases involving complex visual relationships, where existing instance-level approaches lacking fine-grained reasoning struggle to yield meaningful insights. Moreover, current benchmarks are typically tailored to specific algorithms or biased toward image classification, with artificial ground truth that fails to reflect real model failures. To address these limitations, we propose SliceLens, a hypothesis-driven framework that leverages LLMs and VLMs to generate and verify diverse failure hypotheses through grounded visual reasoning, enabling reliable identification of fine-grained and interpretable error slices. We further introduce FeSD (Fine-grained Slice Discovery), the first benchmark specifically designed for evaluating fine-grained error slice discovery across instance-level vision tasks, featuring expert-annotated and carefully refined ground-truth slices with precise grounding to local error regions. Extensive experiments on both existing benchmarks and FeSD demonstrate that SliceLens achieves state-of-the-art performance, improving Precision@10 by 0.42 (0.73 vs. 0.31) on FeSD, and identifies interpretable slices that facilitate actionable model improvements, as validated through model repair experiments.
Abstract (translated)
计算机视觉模型在具有连贯视觉模式的子集上出现系统性错误，这些错误被称为误差切片（error slices），对稳健的模型评估构成了重大挑战。现有的切片发现方法主要针对图像分类任务开发，这限制了它们在检测、分割和姿态估计等多实例任务中的应用范围。在实际场景中，误差切片通常源于涉及复杂视觉关系的边缘情况，在这种情况下，现有仅基于单个实例的方法由于缺乏细粒度推理而难以提供有价值的见解。此外，现有的基准测试通常针对特定算法或偏向于图像分类，并且其人工真值无法准确反映模型的实际错误。 为了克服这些局限性，我们提出了SliceLens这一假设驱动的框架，该框架利用大型语言模型（LLMs）和视觉语言模型（VLMs），通过基于场景的视觉推理生成并验证多样化的失败假设，从而可靠地识别出细粒度且具有解释性的误差切片。此外，我们引入了FeSD (Fine-grained Slice Discovery)，这是首个专门用于评估实例级计算机视觉任务中细粒度错误切片发现性能的基准测试平台，它包括专家注释和精心调整的真实错误区域。 在现有基准测试和FeSD上的广泛实验表明，SliceLens达到了最先进的性能，在FeSD上将Precision@10从0.31提高到了0.73，并且能够识别出解释性很强的误差切片，这些切片通过模型修复实验证明有助于推动有效的模型改进。
URL
https://arxiv.org/abs/2512.24592
PDF
-
MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs
2025-11-18 05:48:08Huiyi Chen, Jiawei Peng, Dehai Min, Changchang Sun, Kaijie Chen, Yan Yan, Xu Yang, Lu ChengAbstract
Evaluating the robustness of Large Vision-Language Models (LVLMs) is essential for their continued development and responsible deployment in real-world applications. However, existing robustness benchmarks typically focus on hallucination or misleading textual inputs, while largely overlooking the equally critical challenge posed by misleading visual inputs in assessing visual understanding. To fill this important gap, we introduce MVI-Bench, the first comprehensive benchmark specially designed for evaluating how Misleading Visual Inputs undermine the robustness of LVLMs. Grounded in fundamental visual primitives, the design of MVI-Bench centers on three hierarchical levels of misleading visual inputs: Visual Concept, Visual Attribute, and Visual Relationship. Using this taxonomy, we curate six representative categories and compile 1,248 expertly annotated VQA instances. To facilitate fine-grained robustness evaluation, we further introduce MVI-Sensitivity, a novel metric that characterizes LVLM robustness at a granular level. Empirical results across 18 state-of-the-art LVLMs uncover pronounced vulnerabilities to misleading visual inputs, and our in-depth analyses on MVI-Bench provide actionable insights that can guide the development of more reliable and robust LVLMs. The benchmark and codebase can be accessed at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.14159
PDF
-
Remodeling Semantic Relationships in Vision-Language Fine-Tuning
2025-11-11 13:37:13Xiangyang Wu, Liu Liu, Baosheng Yu, Jiayan Qiu, Zhenwei ShiAbstract
Vision-language fine-tuning has emerged as an efficient paradigm for constructing multimodal foundation models. While textual context often highlights semantic relationships within an image, existing fine-tuning methods typically overlook this information when aligning vision and language, thus leading to suboptimal performance. Toward solving this problem, we propose a method that can improve multimodal alignment and fusion based on both semantics and this http URL, we first extract multilevel semantic features from different vision encoder to capture more visual cues of the relationships. Then, we learn to project the vision features to group related semantics, among which are more likely to have relationships. Finally, we fuse the visual features with the textual by using inheritable cross-attention, where we globally remove the redundant visual relationships by discarding visual-language feature pairs with low correlation. We evaluate our proposed method on eight foundation models and two downstream tasks, visual question answering and image captioning, and show that it outperforms all existing methods.
Abstract (translated)
URL
https://arxiv.org/abs/2511.08238
PDF
-
VQArt-Bench: A semantically rich VQA Benchmark for Art and Cultural Heritage
2025-10-14 17:29:52A. Alfarano (University of Zurich, Max Planck Society), L. Venturoli (University of Zurich, Max Planck Society), D. Negueruela del Castillo (University of Zurich, Max Planck Society)Abstract
Multimodal Large Language Models (MLLMs) have demonstrated significant capabilities in joint visual and linguistic tasks. However, existing Visual Question Answering (VQA) benchmarks often fail to evaluate deep semantic understanding, particularly in complex domains like visual art analysis. Confined to simple syntactic structures and surface-level attributes, these questions fail to capture the diversity and depth of human visual inquiry. This limitation incentivizes models to exploit statistical shortcuts rather than engage in visual reasoning. To address this gap, we introduce VQArt-Bench, a new, large-scale VQA benchmark for the cultural heritage domain. This benchmark is constructed using a novel multi-agent pipeline where specialized agents collaborate to generate nuanced, validated, and linguistically diverse questions. The resulting benchmark is structured along relevant visual understanding dimensions that probe a model's ability to interpret symbolic meaning, narratives, and complex visual relationships. Our evaluation of 14 state-of-the-art MLLMs on this benchmark reveals significant limitations in current models, including a surprising weakness in simple counting tasks and a clear performance gap between proprietary and open-source models.
Abstract (translated)
多模态大型语言模型（MLLMs）在视觉和语言任务的联合处理方面展示了显著的能力。然而，现有的视觉问答（VQA）基准测试常常无法评估深层语义理解能力，尤其是在像视觉艺术分析这样复杂的领域中。这些问题局限于简单的句法结构和表面属性，未能捕捉到人类视觉查询的多样性和深度。这种局限性促使模型利用统计捷径而不是进行视觉推理。为了解决这一差距，我们引入了VQArt-Bench，这是一个新的大规模VQA基准测试工具，专门针对文化遗产领域设计。该基准测试是通过一个新颖的多代理管道构建的，在这个管道中，专业代理人合作生成细腻、验证过的和语言多样化的提问。由此产生的基准测试涵盖了相关的视觉理解维度，以考察模型解读象征意义、叙述以及复杂视觉关系的能力。我们在这一基准上对14个最先进的MLLMs进行了评估，结果揭示了现有模型中存在的显著局限性，包括在简单的计数任务中的意外薄弱表现，以及专有和开源模型之间的明显性能差距。
URL
https://arxiv.org/abs/2510.12750
PDF
-
B2N3D: Progressive Learning from Binary to N-ary Relationships for 3D Object Grounding
2025-10-11 12:17:12Feng Xiao, Hongbin Xu, Hai Ci, Wenxiong KangAbstract
Localizing 3D objects using natural language is essential for robotic scene understanding. The descriptions often involve multiple spatial relationships to distinguish similar objects, making 3D-language alignment difficult. Current methods only model relationships for pairwise objects, ignoring the global perceptual significance of n-ary combinations in multi-modal relational understanding. To address this, we propose a novel progressive relational learning framework for 3D object grounding. We extend relational learning from binary to n-ary to identify visual relations that match the referential description globally. Given the absence of specific annotations for referred objects in the training data, we design a grouped supervision loss to facilitate n-ary relational learning. In the scene graph created with n-ary relationships, we use a multi-modal network with hybrid attention mechanisms to further localize the target within the n-ary combinations. Experiments and ablation studies on the ReferIt3D and ScanRefer benchmarks demonstrate that our method outperforms the state-of-the-art, and proves the advantages of the n-ary relational perception in 3D localization.
Abstract (translated)
使用自然语言定位3D对象对于机器人场景理解至关重要。这些描述通常涉及多个空间关系来区分相似的对象，从而使得3D和语言之间的对齐变得复杂。目前的方法仅针对成对物体建模其关系，忽略了在多模式关系理解中n-元组合的全局感知重要性。为了解决这个问题，我们提出了一种新颖的渐进式关系学习框架用于3D对象定位。我们将关系学习从二元扩展到n-元，以识别与指称描述相匹配的整体视觉关系。由于训练数据中缺乏具体针对参考对象的注释，我们设计了一个分组监督损失函数来促进n-元关系学习。在使用n-元关系构建的场景图中，我们采用一种具有混合注意力机制的多模式网络进一步在n-元组合内定位目标。我们在ReferIt3D和ScanRefer基准测试上的实验及消融研究证明了我们的方法优于最先进的方法，并且验证了n-元感知关系在3D定位中的优势。
URL
https://arxiv.org/abs/2510.10194
PDF
-
Measuring Image-Relation Alignment: Reference-Free Evaluation of VLMs and Synthetic Pre-training for Open-Vocabulary Scene Graph Generation
2025-09-01 07:46:58Ma\"elic Neau, Zoe Falomir, C\'edric Buche, Akihiro SugimotoAbstract
Scene Graph Generation (SGG) encodes visual relationships between objects in images as graph structures. Thanks to the advances of Vision-Language Models (VLMs), the task of Open-Vocabulary SGG has been recently proposed where models are evaluated on their functionality to learn a wide and diverse range of relations. Current benchmarks in SGG, however, possess a very limited vocabulary, making the evaluation of open-source models inefficient. In this paper, we propose a new reference-free metric to fairly evaluate the open-vocabulary capabilities of VLMs for relation prediction. Another limitation of Open-Vocabulary SGG is the reliance on weakly supervised data of poor quality for pre-training. We also propose a new solution for quickly generating high-quality synthetic data through region-specific prompt tuning of VLMs. Experimental results show that pre-training with this new data split can benefit the generalization capabilities of Open-Voc SGG models.
Abstract (translated)
场景图生成（SGG）任务旨在将图像中物体之间的视觉关系编码为图形结构。得益于视觉-语言模型（VLMs）的进步，最近提出了开放词汇表的SGG任务，评估模型学习广泛且多样化的关系的能力。然而，现有的SGG基准测试具有的词汇量非常有限，这使得开源模型的效果评估变得效率低下。 本文中，我们提出了一种新的无参考度量标准，旨在公平地评估VLM在关系预测中的开放词汇表能力。另外，开放词汇表的SGG任务还受限于依赖质量较差的弱监督数据进行预训练的问题。为此，我们也提出了一个新方案，通过区域特定提示调优（region-specific prompt tuning）的方法快速生成高质量的合成数据。 实验结果表明，使用这种新的数据划分方式进行预训练可以提升开放词汇表SGG模型的泛化能力。
URL
https://arxiv.org/abs/2509.01209
PDF
