Abstract
In remote sensing scene classification, leveraging the transfer methods with well-trained optical models is an efficient way to overcome label scarcity. However, cloud contamination leads to optical information loss and significant impacts on feature distribution, challenging the reliability and stability of transferred target models. Common solutions include cloud removal for optical data or directly using Synthetic aperture radar (SAR) data in the target domain. However, cloud removal requires substantial auxiliary data for support and pre-training, while directly using SAR disregards the unobstructed portions of optical data. This study presents a scene classification transfer method that synergistically combines multi-modality data, which aims to transfer the source domain model trained on cloudfree optical data to the target domain that includes both cloudy optical and SAR data at low cost. Specifically, the framework incorporates two parts: (1) the collaborative transfer strategy, based on knowledge distillation, enables the efficient prior knowledge transfer across heterogeneous data; (2) the information regulation mechanism (IRM) is proposed to address the modality imbalance issue during transfer. It employs auxiliary models to measure the contribution discrepancy of each modality, and automatically balances the information utilization of modalities during the target model learning process at the sample-level. The transfer experiments were conducted on simulated and real cloud datasets, demonstrating the superior performance of the proposed method compared to other solutions in cloud-covered scenarios. We also verified the importance and limitations of IRM, and further discussed and visualized the modality imbalance problem during the model transfer. Codes are available at this https URL
Abstract (translated)
在遥感场景分类中，利用经过良好训练的光学模型进行迁移学习是一种克服标签稀缺的有效方法。然而，云污染会导致光学信息丢失，并对特征分布产生显著影响，从而挑战了转移目标模型的可靠性和稳定性。常见的解决方案包括去除光学数据中的云层或直接使用合成孔径雷达（SAR）数据作为目标领域。但是，云层移除需要大量的辅助数据进行支持和预训练，而直接使用SAR则忽略了未被遮挡部分的光学数据。 本研究提出了一种场景分类迁移方法，该方法协同结合了多模态数据，旨在以低成本将源域中在无云条件下训练好的光学模型迁移到目标领域，其中包含有云的光学数据和SAR数据。具体而言，框架包括两个部分：（1）基于知识蒸馏的合作转移策略，能够有效地跨异构数据进行先验知识迁移；（2）信息调节机制（IRM），用于解决在迁移过程中模态不平衡的问题。该机制采用辅助模型来测量每种模态的贡献差异，并在目标模型学习过程中的样本级别自动平衡各种模态的信息利用。 转移实验是在模拟和真实云数据集上进行的，结果表明，在被云覆盖的情况下，所提出的方法相较于其他解决方案具有优越的表现。我们还验证了IRM的重要性和局限性，并进一步讨论并可视化了在模型迁移过程中出现的模态不平衡问题。代码可在以下链接获取：[此处插入实际URL]
URL
https://arxiv.org/abs/2501.04283
