Abstract
Large Vision and Language Models (LVLMs) have shown strong performance across various vision-language tasks in natural image domains. However, their application to remote sensing (RS) remains underexplored due to significant domain differences in visual appearances, object scales, and semantics. These discrepancies hider the effective understanding of RS scenes, which contain rich, multi-level semantic information spanning from coarse-to-fine levels. Hence, it limits the direct adaptation of existing LVLMs to RS imagery. To address this gap, we propose a novel LVLM framework tailored for RS understanding, incorporating two core components: Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modeling. First, to align multi-level visual features, we introduce the retrieval-based Semantic Augmentation Module which enriches the visual features with relevant semantics across fine-to-coarse levels (e.g., object- and scene-level information). It is designed to retrieve relevant semantic cues from a RS semantic knowledge database, followed by aggregation of semantic cues with user query and multi-level visual features, resulting in semantically enriched representation across multiple levels. Second, for Semantic-aware Expert Modeling, we design semantic experts, where each expert is responsible for processing semantic representation at different levels separately. This enables hierarchical semantic understanding from coarse to fine levels. Evaluations across multiple RS tasks-including scene classification and VQA, etc.-demonstrate that the proposed framework achieves consistent improvements across multiple semantic levels. This highlights its capability and effectiveness in bridging the gap between general LVLMs and unique demands of RS-specific vision-language understanding.
Abstract (translated)
大型视觉和语言模型（LVLM）在自然图像领域的各种视觉-语言任务中表现出色。然而，由于遥感（RS）领域在视觉外观、对象尺度和语义方面存在显著差异，这些模型在遥感中的应用仍较少被探索。这种差距阻碍了对包含从粗到细多层次语义信息的遥感场景的有效理解，从而限制了现有LVLM直接应用于遥感图像的可能性。 为解决这一缺口，我们提出了一种针对遥感理解设计的新框架，该框架融合了两个核心组件：语义增强的多级对齐和感知语义的专家建模。首先，在进行多级视觉特征对准时，我们引入基于检索的语义增强模块，通过从细到粗（如对象级和场景级）层次上与相关语义结合来丰富视觉特征。此模块旨在从遥感语义知识数据库中提取相关的语义线索，并将这些线索与用户查询及多层视觉特征进行聚合，从而生成跨多个层面的语义丰富的表示。 其次，在感知语义专家建模方面，我们设计了专门处理不同层次上语义表示的专家模型。这使从粗到细级别的分层语义理解成为可能。通过在场景分类和VQA等多遥感任务中的评估显示，所提出的框架能够在多个语义层面实现一致性的改进，突显其连接通用LVLM与特定于遥感需求的能力及有效性。
URL
https://arxiv.org/abs/2506.21863
