Abstract
Scene understanding in remote sensing often faces challenges in generating accurate representations for complex environments such as various land use areas or coastal regions, which may also include snow, clouds, or haze. To address this, we present a vision-language framework named Spectral LLaVA, which integrates multispectral data with vision-language alignment techniques to enhance scene representation and description. Using the BigEarthNet v2 dataset from Sentinel-2, we establish a baseline with RGB-based scene descriptions and further demonstrate substantial improvements through the incorporation of multispectral information. Our framework optimizes a lightweight linear projection layer for alignment while keeping the vision backbone of SpectralGPT frozen. Our experiments encompass scene classification using linear probing and language modeling for jointly performing scene classification and description generation. Our results highlight Spectral LLaVA's ability to produce detailed and accurate descriptions, particularly for scenarios where RGB data alone proves inadequate, while also enhancing classification performance by refining SpectralGPT features into semantically meaningful representations.
Abstract (translated)
遥感中的场景理解常常面临挑战，尤其是在生成复杂环境（如各种土地利用区域或沿海地区）的准确表示时，这些环境中可能还包括雪、云或雾霾等干扰因素。为了解决这些问题，我们提出了一种视觉-语言框架，名为Spectral LLaVA，该框架将多光谱数据与视觉-语言对齐技术相结合，以增强场景的表现和描述能力。使用Sentinel-2卫星的BigEarthNet v2数据集，我们在基于RGB图像的场景描述基础上建立了基准，并通过引入多光谱信息进一步展示了显著改进。 我们的框架优化了一层轻量级线性投影层来实现视觉与语言的对齐，同时保持SpectralGPT的视觉骨干网络不变。实验涵盖了使用线性探测进行场景分类以及联合执行场景分类和描述生成的语言建模任务。我们的结果突显了Spectral LLaVA在产生详细且准确描述方面的能力，特别是在仅依靠RGB数据不足以充分描述的情况下表现尤为突出，并通过将SpectralGPT的特征精炼为语义上有意义的表现来提高分类性能。 简而言之，Spectral LLaVA框架不仅能够生成对复杂环境场景的细致和精确描述，还能够在多光谱图像处理中提升模型在场景理解任务中的效果。
URL
https://arxiv.org/abs/2501.10144
