Abstract
The domain gap between remote sensing imagery and natural images has recently received widespread attention and Vision-Language Models (VLMs) have demonstrated excellent generalization performance in remote sensing multimodal tasks. However, current research is still limited in exploring how remote sensing VLMs handle different types of visual inputs. To bridge this gap, we introduce \textbf{UniRS}, the first vision-language model \textbf{uni}fying multi-temporal \textbf{r}emote \textbf{s}ensing tasks across various types of visual input. UniRS supports single images, dual-time image pairs, and videos as input, enabling comprehensive remote sensing temporal analysis within a unified framework. We adopt a unified visual representation approach, enabling the model to accept various visual inputs. For dual-time image pair tasks, we customize a change extraction module to further enhance the extraction of spatiotemporal features. Additionally, we design a prompt augmentation mechanism tailored to the model's reasoning process, utilizing the prior knowledge of the general-purpose VLM to provide clues for UniRS. To promote multi-task knowledge sharing, the model is jointly fine-tuned on a mixed dataset. Experimental results show that UniRS achieves state-of-the-art performance across diverse tasks, including visual question answering, change captioning, and video scene classification, highlighting its versatility and effectiveness in unifying these multi-temporal remote sensing tasks. Our code and dataset will be released soon.
Abstract (translated)
最近，遥感图像与自然图像之间的领域差距引起了广泛的关注，并且视觉-语言模型（VLMs）在多模态遥感任务中展示了出色的泛化性能。然而，目前的研究仍然局限于探索遥感 VLM 如何处理不同类型的数据输入。为了弥补这一不足，我们引入了 **UniRS**，这是第一个统一跨多种类型视觉输入的多时态遥感任务的视觉-语言模型。UniRS 支持单幅图像、双时间图像对和视频作为输入，在一个统一框架内实现了全面的遥感时态分析。我们采用了一种统一的视觉表示方法，使模型能够接受各种视觉输入。对于双时间图像对任务，我们定制了一个变化提取模块，以进一步增强空间-时间特征的抽取能力。此外，我们设计了一种针对模型推理过程优化的提示增强机制，利用通用 VLM 的先验知识为 UniRS 提供线索。为了促进多任务的知识共享，该模型在混合数据集上进行了联合微调。实验结果表明，UniRS 在视觉问答、变化描述和视频场景分类等多种任务中均达到了最先进的性能，突显了其在统一这些多时态遥感任务中的多功能性和有效性。我们的代码和数据集将很快发布。
URL
https://arxiv.org/abs/2412.20742
