Abstract
Recent advances in self-supervised learning for Vision Transformers (ViTs) have fueled breakthroughs in remote sensing (RS) foundation models. However, the quadratic complexity of self-attention poses a significant barrier to scalability, particularly for large models and high-resolution images. While the linear-complexity Mamba architecture offers a promising alternative, existing RS applications of Mamba remain limited to supervised tasks on small, domain-specific datasets. To address these challenges, we propose RoMA, a framework that enables scalable self-supervised pretraining of Mamba-based RS foundation models using large-scale, diverse, unlabeled data. RoMA enhances scalability for high-resolution images through a tailored auto-regressive learning strategy, incorporating two key innovations: 1) a rotation-aware pretraining mechanism combining adaptive cropping with angular embeddings to handle sparsely distributed objects with arbitrary orientations, and 2) multi-scale token prediction objectives that address the extreme variations in object scales inherent to RS imagery. Systematic empirical studies validate that Mamba adheres to RS data and parameter scaling laws, with performance scaling reliably as model and data size increase. Furthermore, experiments across scene classification, object detection, and semantic segmentation tasks demonstrate that RoMA-pretrained Mamba models consistently outperform ViT-based counterparts in both accuracy and computational efficiency. The source code and pretrained models will be released at this https URL.
Abstract (translated)
最近在视觉变换器（ViTs）的自监督学习方面的进展推动了远程传感（RS）基础模型的重大突破。然而，自我注意机制的二次复杂性构成了扩展性的重大障碍，尤其是对于大型模型和高分辨率图像而言。虽然线性复杂度的Mamba架构提供了一个有前景的替代方案，但现有的基于Mamba的RS应用仍然局限于小规模、特定领域的监督任务上。为了解决这些挑战，我们提出了RoMA框架，该框架可以利用大规模多样化的未标记数据进行基于Mamba的RS基础模型的大规模自监督预训练。通过定制的自回归学习策略，RoMA提升了对高分辨率图像的扩展性，并引入了两项关键创新： 1. 一种旋转感知的预训练机制，结合了适应性裁剪与角向嵌入，以处理具有任意方向的稀疏分布的对象。 2. 多尺度标记预测目标，解决了RS影像中固有的物体尺寸极端变化问题。 系统性的实证研究表明，Mamba遵循了适用于RS数据和参数缩放规律，并且随着模型和数据规模的增长，其性能可以可靠地扩展。此外，在场景分类、对象检测和语义分割任务中的实验表明，RoMA预训练的Mamba模型在精度和计算效率方面均优于基于ViT的同类模型。 相关源代码和预训练模型将在以下网址发布：[此链接]（请根据实际情况提供实际链接）。
URL
https://arxiv.org/abs/2503.10392
