Abstract
Few-Shot Remote Sensing Scene Classification (FS-RSSC) presents the challenge of classifying remote sensing images with limited labeled samples. Existing methods typically emphasize single-modal feature learning, neglecting the potential benefits of optimizing multi-modal representations. To address this limitation, we propose a novel Optimal Transport Adapter Tuning (OTAT) framework aimed at constructing an ideal Platonic representational space through optimal transport (OT) theory. This framework seeks to harmonize rich visual information with less dense textual cues, enabling effective cross-modal information transfer and complementarity. Central to this approach is the Optimal Transport Adapter (OTA), which employs a cross-modal attention mechanism to enrich textual representations and facilitate subsequent better information interaction. By transforming the network optimization into an OT optimization problem, OTA establishes efficient pathways for balanced information exchange between modalities. Moreover, we introduce a sample-level Entropy-Aware Weighted (EAW) loss, which combines difficulty-weighted similarity scores with entropy-based regularization. This loss function provides finer control over the OT optimization process, enhancing its solvability and stability. Our framework offers a scalable and efficient solution for advancing multimodal learning in remote sensing applications. Extensive experiments on benchmark datasets demonstrate that OTAT achieves state-of-the-art performance in FS-RSSC, significantly improving the model performance and generalization.
Abstract (translated)
少样本遥感场景分类（Few-Shot Remote Sensing Scene Classification，FS-RSSC）面临的主要挑战是在仅有少量标注样本的情况下对遥感图像进行准确分类。现有的方法通常侧重于单一模态特征学习，忽视了优化多模态表示的潜在优势。为了解决这一局限性，我们提出了一种新颖的最优传输适配器调优（Optimal Transport Adapter Tuning，OTAT）框架，旨在通过最优传输理论构建理想化的柏拉图式表征空间。该框架致力于协调丰富的视觉信息与较少密度的文字线索，从而实现有效的跨模态信息传递和互补性。这一方法的核心是最佳传输适配器（Optimal Transport Adapter, OTA），它采用跨模态注意力机制来丰富文本表示，并促进后续更好的信息交互。通过将网络优化转化为最优传输优化问题，OTA建立了模态间平衡的信息交换高效路径。此外，我们还引入了一种样本级别的熵感知加权（Entropy-Aware Weighted，EAW）损失函数，该函数结合了基于难度的相似度得分和基于熵的正则化项。这种损失函数对OT优化过程提供了更精细的控制，增强了其可解性和稳定性。我们的框架为推进遥感应用中的多模态学习提供了一种可扩展且高效的解决方案。在基准数据集上的广泛实验表明，OTAT在FS-RSSC任务中达到了最先进的性能，并显著提升了模型的表现和泛化能力。
URL
https://arxiv.org/abs/2503.14938
