Abstract
The application of Vision-Language Models (VLMs) in remote sensing (RS) has demonstrated significant potential in traditional tasks such as scene classification, object detection, and image captioning. However, current models, which excel in Referring Expression Comprehension (REC), struggle with tasks involving complex instructions (e.g., exists multiple conditions) or pixel-level operations like segmentation and change detection. In this white paper, we provide a comprehensive hierarchical summary of vision-language tasks in RS, categorized by the varying levels of cognitive capability required. We introduce the Remote Sensing Vision-Language Task Set (RSVLTS), which includes Open-Vocabulary Tasks (OVT), Referring Expression Tasks (RET), and Described Object Tasks (DOT) with increased difficulty, and Visual Question Answering (VQA) aloneside. Moreover, we propose a novel unified data representation using a set-of-points approach for RSVLTS, along with a condition parser and a self-augmentation strategy based on cyclic referring. These features are integrated into the GeoRSMLLM model, and this enhanced model is designed to handle a broad range of tasks of RSVLTS, paving the way for a more generalized solution for vision-language tasks in geoscience and remote sensing.
Abstract (translated)
在遥感（RS）领域中，视觉-语言模型（VLMs）的应用已在场景分类、目标检测和图像描述等传统任务上展示了显著的潜力。然而，目前擅长于指称表达理解（REC）的任务模型，在处理涉及复杂指令（例如存在多个条件的情况）或像素级操作（如分割和变化检测）的任务时却显得力不从心。 在本白皮书中，我们提供了一套全面分层总结的遥感视觉-语言任务，并根据所需认知能力的不同层次进行分类。我们引入了远程感知视觉-语言任务集（RSVLTS），该任务集包括开放词汇任务（OVT）、指称表达任务（RET）和描述对象任务（DOT），难度依次递增，同时涵盖了视觉问答（VQA）。此外，我们提出了一种新颖的统一数据表示方法，采用点集方式来处理RSVLTS，并引入了一个条件解析器以及基于循环指代的自我增强策略。 这些特征被整合到GeoRSMLLM模型中，该改进后的模型旨在应对RSVLTS广泛任务范围的需求，为地理科学和遥感领域的视觉-语言任务提供更加通用的解决方案。
URL
https://arxiv.org/abs/2503.12490
