Abstract
Remote sensing data is often distributed across multiple institutions, and due to privacy concerns and data-sharing restrictions, leveraging large-scale datasets in a centralized training framework is challenging. Federated learning offers a promising solution by enabling collaborative model training across distributed data sources without requiring data centralization. However, current Vision-Language Models (VLMs), which typically contain billions of parameters, pose significant communication challenges for traditional federated learning approaches based on model parameter updates, as they would incur substantial communication costs. In this paper, we propose FedRSCLIP, the first federated learning framework designed for remote sensing image classification based on a VLM, specifically CLIP. FedRSCLIP addresses the challenges of data heterogeneity and large-scale model transmission in federated environments by introducing Prompt Learning, which optimizes only a small set of tunable parameters. The framework introduces a dual-prompt mechanism, comprising Shared Prompts for global knowledge sharing and Private Prompts for client-specific adaptation. To maintain semantic coherence between shared and private prompts, we propose the Dual Prompt Alignment Constraint to balance global consistency and local adaptability across diverse client distributions. Additionally, to enhance cross-modal representation learning, we introduce the Cross-Modal Feature Alignment Constraint to align multimodal features between text and image prompts. To validate the effectiveness of our proposed model, we construct a Fed-RSIC dataset based on three existing remote sensing image classification datasets, specifically designed to simulate various federated learning configurations. Experimental results demonstrate the effectiveness and superiority of FedRSCLIP in remote sensing image classification.
Abstract (translated)
遥感数据通常分布在多个机构之间，由于隐私顾虑和数据共享限制，在集中式训练框架中利用大规模的数据集具有挑战性。联邦学习通过允许多个分布式数据源协同模型训练而不需进行数据集中化提供了一个有前景的解决方案。然而，现有的视觉-语言模型（VLMs）通常包含数十亿参数，这对基于模型参数更新的传统联邦学习方法提出了显著的通信挑战，因为这将带来高昂的通信成本。在本文中，我们提出FedRSCLIP，这是首个为基于视觉-语言模型CLIP的遥感图像分类设计的联邦学习框架。FedRSCLIP通过引入提示学习解决了数据异构性和大规模模型传输在联邦环境中的挑战，仅优化一小部分可调参数。该框架引入了双提示机制，包括用于全局知识共享的共享提示和针对特定客户端适应的私有提示。为了保持共享与私人提示之间的语义一致性，我们提出了双提示对齐约束以在全球一致性和局部适应性之间平衡各种客户端分布情况下的差异。此外，为增强跨模态表示学习，我们引入了跨模态特征对齐约束，用于调整文本和图像提示之间的多模式特征。为了验证所提出模型的有效性，我们基于三个现有的遥感图像分类数据集构建了一个Fed-RSIC数据集，专门设计以模拟各种联邦学习配置。实验结果证明了FedRSCLIP在遥感图像分类中的有效性和优越性。
URL
https://arxiv.org/abs/2501.02461
