Abstract
Vision-language models (VLMs) have demonstrated impressive capabilities in understanding and reasoning about visual and textual content. However, their robustness to common image corruptions remains under-explored. In this work, we present the first comprehensive analysis of VLM robustness across 19 corruption types from the ImageNet-C benchmark, spanning four categories: noise, blur, weather, and digital distortions. We introduce two new benchmarks, TextVQA-C and GQA-C, to systematically evaluate how corruptions affect scene text understanding and object-based reasoning, respectively. Our analysis reveals that transformer-based VLMs exhibit distinct vulnerability patterns across tasks: text recognition deteriorates most severely under blur and snow corruptions, while object reasoning shows higher sensitivity to corruptions such as frost and impulse noise. We connect these observations to the frequency-domain characteristics of different corruptions, revealing how transformers' inherent bias toward low-frequency processing explains their differential robustness patterns. Our findings provide valuable insights for developing more corruption-robust vision-language models for real-world applications.
Abstract (translated)
视觉语言模型（VLMs）在理解和推理视觉和文本内容方面表现出令人印象深刻的能力。然而，它们对常见图像损坏的鲁棒性仍鲜有研究。在这项工作中，我们首次全面分析了来自ImageNet-C基准测试的19种不同类型图像损坏对VLM鲁棒性的影响，这些类型涵盖了四个类别：噪声、模糊、天气和数字扭曲。我们介绍了两个新的基准测试——TextVQA-C和GQA-C，分别系统地评估图像损坏如何影响场景文本理解和基于对象的推理。我们的分析揭示了基于变压器的VLM在不同任务中表现出独特的脆弱性模式：文本识别在遭受模糊和雪破坏时退化最为严重；而物体推理则对霜冻和脉冲噪声等破坏表现出更高的敏感度。我们将这些观察结果与不同类型损坏的频域特性联系起来，展示了变压器模型固有的低频处理偏见如何解释其不同的鲁棒性模式。我们的发现为开发更适用于现实世界应用、能更好地抵抗图像损坏影响的视觉语言模型提供了宝贵的见解。
URL
https://arxiv.org/abs/2504.13690
