Scene_Classification
-
What Time Tells Us? An Explorative Study of Time Awareness Learned from Static Images
2025-03-23 01:56:35Dongheng Lin, Han Hu, Jianbo JiaoAbstract
Time becomes visible through illumination changes in what we see. Inspired by this, in this paper we explore the potential to learn time awareness from static images, trying to answer: what time tells us? To this end, we first introduce a Time-Oriented Collection (TOC) dataset, which contains 130,906 images with reliable timestamps. Leveraging this dataset, we propose a Time-Image Contrastive Learning (TICL) approach to jointly model timestamps and related visual representations through cross-modal contrastive learning. We found that the proposed TICL, 1) not only achieves state-of-the-art performance on the timestamp estimation task, over various benchmark metrics, 2) but also, interestingly, though only seeing static images, the time-aware embeddings learned from TICL show strong capability in several time-aware downstream tasks such as time-based image retrieval, video scene classification, and time-aware image editing. Our findings suggest that time-related visual cues can be learned from static images and are beneficial for various vision tasks, laying a foundation for future research on understanding time-related visual context. Project page:this https URL.
Abstract (translated)
时间通过我们所见事物的照明变化而变得可见。受此启发，本文探讨了从静态图像中学习时间感知的潜力，并尝试回答：“时间告诉我们什么？”为此，我们首先引入了一个以时间为方向的数据集（Time-Oriented Collection，简称TOC），该数据集中包含130,906张带有可靠时间戳的照片。利用这一数据集，我们提出了一种时间-图像对比学习（Time-Image Contrastive Learning，简称TICL）方法，通过跨模态对比学习共同建模时间戳及其相关的视觉表示形式。我们发现，所提出的TICL方法不仅在时间戳估计任务上实现了最先进的性能，而且令人惊讶的是，在仅接触静态图像的情况下，从TICL中学习到的时间感知嵌入在时间感知下游任务（如基于时间的图片检索、视频场景分类和具有时间意识的图片编辑）方面表现出很强的能力。我们的发现表明，可以从静态图像中学习与时间相关的视觉线索，并且这对各种视觉任务有益，为未来研究理解与时间相关的情境奠定了基础。项目页面：[此链接](https://此链接/)。
URL
https://arxiv.org/abs/2503.17899
PDF
-
City2Scene: Improving Acoustic Scene Classification with City Features
2025-03-21 05:24:48Yiqiang Cai, Yizhou Tan, Peihong Zhang, Yuxuan Liu, Shengchen Li, Xi Shao, Mark D. PlumbleyAbstract
Acoustic scene recordings are often collected from a diverse range of cities. Most existing acoustic scene classification (ASC) approaches focus on identifying common acoustic scene patterns across cities to enhance generalization. In contrast, we hypothesize that city-specific environmental and cultural differences in acoustic features are beneficial for the ASC task. In this paper, we introduce City2Scene, a novel framework that leverages city features to improve ASC. City2Scene transfers the city-specific knowledge from city classification models to a scene classification model using knowledge distillation. We evaluated City2Scene on the DCASE Challenge Task 1 datasets, where each audio clip is annotated with both scene and city labels. Experimental results demonstrate that city features provide valuable information for classifying scenes. By distilling the city-specific knowledge, City2Scene effectively improves accuracy for various state-of-the-art ASC backbone models, including both CNNs and Transformers.
Abstract (translated)
声学场景录音通常是从不同城市的多样化环境中收集的。大多数现有的声学场景分类（ASC）方法侧重于识别跨城市之间普遍存在的声学场景模式，以增强泛化能力。相比之下，我们假设每个城市特有的环境和文化差异在声学特征中是有利于ASC任务的。本文介绍了City2Scene，这是一种新的框架，利用城市的特性来改进ASC。City2Scene通过知识蒸馏将特定于城市的知识从城市分类模型转移到场景分类模型上。 我们在DCASE挑战赛任务1数据集上评估了City2Scene，该数据集中每个音频片段都带有声学场景和城市的标签。实验结果表明，城市特征为声学场景的分类提供了有价值的信息。通过蒸馏出特定于城市的知识，City2Scene有效地提高了各种先进的ASC骨干模型（包括CNNs和Transformers）的准确性。
URL
https://arxiv.org/abs/2503.16862
PDF
-
Optimal Transport Adapter Tuning for Bridging Modality Gaps in Few-Shot Remote Sensing Scene Classification
2025-03-19 07:04:24Zhong Ji, Ci Liu, Jingren Liu, Chen Tang, Yanwei Pang, Xuelong LiAbstract
Few-Shot Remote Sensing Scene Classification (FS-RSSC) presents the challenge of classifying remote sensing images with limited labeled samples. Existing methods typically emphasize single-modal feature learning, neglecting the potential benefits of optimizing multi-modal representations. To address this limitation, we propose a novel Optimal Transport Adapter Tuning (OTAT) framework aimed at constructing an ideal Platonic representational space through optimal transport (OT) theory. This framework seeks to harmonize rich visual information with less dense textual cues, enabling effective cross-modal information transfer and complementarity. Central to this approach is the Optimal Transport Adapter (OTA), which employs a cross-modal attention mechanism to enrich textual representations and facilitate subsequent better information interaction. By transforming the network optimization into an OT optimization problem, OTA establishes efficient pathways for balanced information exchange between modalities. Moreover, we introduce a sample-level Entropy-Aware Weighted (EAW) loss, which combines difficulty-weighted similarity scores with entropy-based regularization. This loss function provides finer control over the OT optimization process, enhancing its solvability and stability. Our framework offers a scalable and efficient solution for advancing multimodal learning in remote sensing applications. Extensive experiments on benchmark datasets demonstrate that OTAT achieves state-of-the-art performance in FS-RSSC, significantly improving the model performance and generalization.
Abstract (translated)
少样本遥感场景分类（Few-Shot Remote Sensing Scene Classification，FS-RSSC）面临的主要挑战是在仅有少量标注样本的情况下对遥感图像进行准确分类。现有的方法通常侧重于单一模态特征学习，忽视了优化多模态表示的潜在优势。为了解决这一局限性，我们提出了一种新颖的最优传输适配器调优（Optimal Transport Adapter Tuning，OTAT）框架，旨在通过最优传输理论构建理想化的柏拉图式表征空间。该框架致力于协调丰富的视觉信息与较少密度的文字线索，从而实现有效的跨模态信息传递和互补性。这一方法的核心是最佳传输适配器（Optimal Transport Adapter, OTA），它采用跨模态注意力机制来丰富文本表示，并促进后续更好的信息交互。通过将网络优化转化为最优传输优化问题，OTA建立了模态间平衡的信息交换高效路径。此外，我们还引入了一种样本级别的熵感知加权（Entropy-Aware Weighted，EAW）损失函数，该函数结合了基于难度的相似度得分和基于熵的正则化项。这种损失函数对OT优化过程提供了更精细的控制，增强了其可解性和稳定性。我们的框架为推进遥感应用中的多模态学习提供了一种可扩展且高效的解决方案。在基准数据集上的广泛实验表明，OTAT在FS-RSSC任务中达到了最先进的性能，并显著提升了模型的表现和泛化能力。
URL
https://arxiv.org/abs/2503.14938
PDF
-
GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing
2025-03-16 12:48:17Zilun Zhang, Haozhan Shen, Tiancheng Zhao, Bin Chen, Zian Guan, Yuhao Wang, Xu Jia, Yuxiang Cai, Yongheng Shang, Jianwei YinAbstract
The application of Vision-Language Models (VLMs) in remote sensing (RS) has demonstrated significant potential in traditional tasks such as scene classification, object detection, and image captioning. However, current models, which excel in Referring Expression Comprehension (REC), struggle with tasks involving complex instructions (e.g., exists multiple conditions) or pixel-level operations like segmentation and change detection. In this white paper, we provide a comprehensive hierarchical summary of vision-language tasks in RS, categorized by the varying levels of cognitive capability required. We introduce the Remote Sensing Vision-Language Task Set (RSVLTS), which includes Open-Vocabulary Tasks (OVT), Referring Expression Tasks (RET), and Described Object Tasks (DOT) with increased difficulty, and Visual Question Answering (VQA) aloneside. Moreover, we propose a novel unified data representation using a set-of-points approach for RSVLTS, along with a condition parser and a self-augmentation strategy based on cyclic referring. These features are integrated into the GeoRSMLLM model, and this enhanced model is designed to handle a broad range of tasks of RSVLTS, paving the way for a more generalized solution for vision-language tasks in geoscience and remote sensing.
Abstract (translated)
在遥感（RS）领域中，视觉-语言模型（VLMs）的应用已在场景分类、目标检测和图像描述等传统任务上展示了显著的潜力。然而，目前擅长于指称表达理解（REC）的任务模型，在处理涉及复杂指令（例如存在多个条件的情况）或像素级操作（如分割和变化检测）的任务时却显得力不从心。 在本白皮书中，我们提供了一套全面分层总结的遥感视觉-语言任务，并根据所需认知能力的不同层次进行分类。我们引入了远程感知视觉-语言任务集（RSVLTS），该任务集包括开放词汇任务（OVT）、指称表达任务（RET）和描述对象任务（DOT），难度依次递增，同时涵盖了视觉问答（VQA）。此外，我们提出了一种新颖的统一数据表示方法，采用点集方式来处理RSVLTS，并引入了一个条件解析器以及基于循环指代的自我增强策略。 这些特征被整合到GeoRSMLLM模型中，该改进后的模型旨在应对RSVLTS广泛任务范围的需求，为地理科学和遥感领域的视觉-语言任务提供更加通用的解决方案。
URL
https://arxiv.org/abs/2503.12490
PDF
-
Creating a Good Teacher for Knowledge Distillation in Acoustic Scene Classification
2025-03-14 12:57:12Tobias Morocutti, Florian Schmid, Khaled Koutini, Gerhard WidmerAbstract
Knowledge Distillation (KD) is a widespread technique for compressing the knowledge of large models into more compact and efficient models. KD has proved to be highly effective in building well-performing low-complexity Acoustic Scene Classification (ASC) systems and was used in all the top-ranked submissions to this task of the annual DCASE challenge in the past three years. There is extensive research available on establishing the KD process, designing efficient student models, and forming well-performing teacher ensembles. However, less research has been conducted on investigating which teacher model attributes are beneficial for low-complexity students. In this work, we try to close this gap by studying the effects on the student's performance when using different teacher network architectures, varying the teacher model size, training them with different device generalization methods, and applying different ensembling strategies. The results show that teacher model sizes, device generalization methods, the ensembling strategy and the ensemble size are key factors for a well-performing student network.
Abstract (translated)
知识蒸馏（KD）是一种将大型模型的知识压缩到更紧凑、高效的模型中的广泛技术。在过去的三年里，KD已被证明是构建高性能低复杂度的声场景分类（ASC）系统的一种非常有效的方法，并且被年度DCASE挑战赛中该任务的所有顶级提交所采用。已有大量研究探讨了建立蒸馏过程、设计高效的学生模型以及形成表现良好的教师模型组合的方法。然而，关于探究哪些教师模型属性对低复杂度学生有益的研究较少。在这项工作中，我们尝试填补这一空白，通过使用不同的教师网络架构、改变教师模型大小、用不同的设备泛化方法对其进行训练，并应用不同的集合策略来研究对学生性能的影响。结果表明，教师模型的规模、设备泛化方法、组合策略和组合大小是构建表现良好的学生网络的关键因素。
URL
https://arxiv.org/abs/2503.11363
PDF
-
APLA: A Simple Adaptation Method for Vision Transformers
2025-03-14 12:03:29Moein Sorkhei, Emir Konuk, Kevin Smith, Christos MatsoukasAbstract
Existing adaptation techniques typically require architectural modifications or added parameters, leading to high computational costs and complexity. We introduce Attention Projection Layer Adaptation (APLA), a simple approach to adapt vision transformers (ViTs) without altering the architecture or adding parameters. Through a systematic analysis, we find that the layer immediately after the attention mechanism is crucial for adaptation. By updating only this projection layer, or even just a random subset of this layer's weights, APLA achieves state-of-the-art performance while reducing GPU memory usage by up to 52.63% and training time by up to 43.0%, with no extra cost at inference. Across 46 datasets covering a variety of tasks including scene classification, medical imaging, satellite imaging, and fine-grained classification, APLA consistently outperforms 17 other leading adaptation methods, including full fine-tuning, on classification, segmentation, and detection tasks. The code is available at this https URL.
Abstract (translated)
现有的适应技术通常需要进行架构修改或添加参数，这导致了高昂的计算成本和复杂性。我们引入了一种简单的方法——注意力投影层适配（Attention Projection Layer Adaptation, APLA），这种方法可以对视觉变换器（Vision Transformers, ViTs）进行调整而无需更改其架构或增加额外的参数。通过系统的分析，我们发现紧随注意机制之后的那一层对于适应至关重要。仅更新这一层中的投影层，甚至只是其中权重的一个随机子集，APLA就能达到最先进的性能，并且在不增加推理成本的情况下，最多可以减少52.63%的GPU内存使用和43.0%的训练时间。在涵盖场景分类、医学成像、卫星成像以及细粒度分类等任务的46个数据集中，APLA在分类、分割和检测任务上始终优于包括全量微调在内的17种其他领先适应方法。代码可在[此处](https://example.com/code)获取。
URL
https://arxiv.org/abs/2503.11335
PDF
-
MEET: A Million-Scale Dataset for Fine-Grained Geospatial Scene Classification with Zoom-Free Remote Sensing Imagery
2025-03-14 09:10:45Yansheng Li, Yuning Wu, Gong Cheng, Chao Tao, Bo Dang, Yu Wang, Jiahao Zhang, Chuge Zhang, Yiting Liu, Xu Tang, Jiayi Ma, Yongjun ZhangAbstract
Accurate fine-grained geospatial scene classification using remote sensing imagery is essential for a wide range of applications. However, existing approaches often rely on manually zooming remote sensing images at different scales to create typical scene samples. This approach fails to adequately support the fixed-resolution image interpretation requirements in real-world scenarios. To address this limitation, we introduce the Million-scale finE-grained geospatial scEne classification dataseT (MEET), which contains over 1.03 million zoom-free remote sensing scene samples, manually annotated into 80 fine-grained categories. In MEET, each scene sample follows a scene-inscene layout, where the central scene serves as the reference, and auxiliary scenes provide crucial spatial context for finegrained classification. Moreover, to tackle the emerging challenge of scene-in-scene classification, we present the Context-Aware Transformer (CAT), a model specifically designed for this task, which adaptively fuses spatial context to accurately classify the scene samples. CAT adaptively fuses spatial context to accurately classify the scene samples by learning attentional features that capture the relationships between the center and auxiliary scenes. Based on MEET, we establish a comprehensive benchmark for fine-grained geospatial scene classification, evaluating CAT against 11 competitive baselines. The results demonstrate that CAT significantly outperforms these baselines, achieving a 1.88% higher balanced accuracy (BA) with the Swin-Large backbone, and a notable 7.87% improvement with the Swin-Huge backbone. Further experiments validate the effectiveness of each module in CAT and show the practical applicability of CAT in the urban functional zone mapping. The source code and dataset will be publicly available at this https URL.
Abstract (translated)
使用遥感图像进行精确的细粒度地理场景分类对于广泛的应用至关重要。然而，现有的方法往往依赖于手动缩放遥感图像以不同比例创建典型的场景样本，这种方法无法充分支持现实世界中固定分辨率图像解释的需求。为了解决这一局限性，我们引入了百万级细粒度地理场景分类数据集（MEET），该数据集包含超过103万张无需缩放的遥感场景样本，并由人工标注到80个细粒度类别中。在MEET数据集中，每个场景样本遵循场景内场景布局，其中中心场景作为参考，辅助场景提供对精细分类至关重要的空间上下文信息。 此外，为了应对场景内场景分类这一新兴挑战，我们提出了一个专门为该任务设计的模型——上下文感知变换器（CAT），它能够自适应地融合空间上下文以准确分类场景样本。CAT通过学习捕捉中心和辅助场景之间关系的关注特征来自适应地融合空间上下文信息。 基于MEET数据集，我们在细粒度地理场景分类方面建立了全面的基准测试，并将CAT与11个具有竞争力的基础模型进行了比较。结果表明，CAT显著优于这些基础模型，在使用Swin-Large骨干网络时，平衡准确率（BA）提高了1.88%，而在使用Swin-Huge骨干网络时，取得了7.87%的显着改进。 进一步的实验验证了CAT中每个模块的有效性，并展示了CAT在城市功能区制图中的实际应用价值。源代码和数据集将在以下网址公开发布：[https URL]。
URL
https://arxiv.org/abs/2503.11219
PDF
-
RoMA: Scaling up Mamba-based Foundation Models for Remote Sensing
2025-03-13 14:09:18Fengxiang Wang, Hongzhen Wang, Yulin Wang, Di Wang, Mingshuo Chen, Haiyan Zhao, Yangang Sun, Shuo Wang, Long Lan, Wenjing Yang, Jing ZhangAbstract
Recent advances in self-supervised learning for Vision Transformers (ViTs) have fueled breakthroughs in remote sensing (RS) foundation models. However, the quadratic complexity of self-attention poses a significant barrier to scalability, particularly for large models and high-resolution images. While the linear-complexity Mamba architecture offers a promising alternative, existing RS applications of Mamba remain limited to supervised tasks on small, domain-specific datasets. To address these challenges, we propose RoMA, a framework that enables scalable self-supervised pretraining of Mamba-based RS foundation models using large-scale, diverse, unlabeled data. RoMA enhances scalability for high-resolution images through a tailored auto-regressive learning strategy, incorporating two key innovations: 1) a rotation-aware pretraining mechanism combining adaptive cropping with angular embeddings to handle sparsely distributed objects with arbitrary orientations, and 2) multi-scale token prediction objectives that address the extreme variations in object scales inherent to RS imagery. Systematic empirical studies validate that Mamba adheres to RS data and parameter scaling laws, with performance scaling reliably as model and data size increase. Furthermore, experiments across scene classification, object detection, and semantic segmentation tasks demonstrate that RoMA-pretrained Mamba models consistently outperform ViT-based counterparts in both accuracy and computational efficiency. The source code and pretrained models will be released at this https URL.
Abstract (translated)
最近在视觉变换器（ViTs）的自监督学习方面的进展推动了远程传感（RS）基础模型的重大突破。然而，自我注意机制的二次复杂性构成了扩展性的重大障碍，尤其是对于大型模型和高分辨率图像而言。虽然线性复杂度的Mamba架构提供了一个有前景的替代方案，但现有的基于Mamba的RS应用仍然局限于小规模、特定领域的监督任务上。为了解决这些挑战，我们提出了RoMA框架，该框架可以利用大规模多样化的未标记数据进行基于Mamba的RS基础模型的大规模自监督预训练。通过定制的自回归学习策略，RoMA提升了对高分辨率图像的扩展性，并引入了两项关键创新： 1. 一种旋转感知的预训练机制，结合了适应性裁剪与角向嵌入，以处理具有任意方向的稀疏分布的对象。 2. 多尺度标记预测目标，解决了RS影像中固有的物体尺寸极端变化问题。 系统性的实证研究表明，Mamba遵循了适用于RS数据和参数缩放规律，并且随着模型和数据规模的增长，其性能可以可靠地扩展。此外，在场景分类、对象检测和语义分割任务中的实验表明，RoMA预训练的Mamba模型在精度和计算效率方面均优于基于ViT的同类模型。 相关源代码和预训练模型将在以下网址发布：[此链接]（请根据实际情况提供实际链接）。
URL
https://arxiv.org/abs/2503.10392
PDF
-
Dual Classification Head Self-training Network for Cross-scene Hyperspectral Image Classification
2025-02-25 06:07:46Rong Liu, Junye Liang, Jiaqi Yang, Jiang He, Peng ZhuAbstract
Due to the difficulty of obtaining labeled data for hyperspectral images (HSIs), cross-scene classification has emerged as a widely adopted approach in the remote sensing community. It involves training a model using labeled data from a source domain (SD) and unlabeled data from a target domain (TD), followed by inferencing on the TD. However, variations in the reflectance spectrum of the same object between the SD and the TD, as well as differences in the feature distribution of the same land cover class, pose significant challenges to the performance of cross-scene classification. To address this issue, we propose a dual classification head self-training network (DHSNet). This method aligns class-wise features across domains, ensuring that the trained classifier can accurately classify TD data of different classes. We introduce a dual classification head self-training strategy for the first time in the cross-scene HSI classification field. The proposed approach mitigates domain gap while preventing the accumulation of incorrect pseudo-labels in the model. Additionally, we incorporate a novel central feature attention mechanism to enhance the model's capacity to learn scene-invariant features across domains. Experimental results on three cross-scene HSI datasets demonstrate that the proposed DHSNET significantly outperforms other state-of-the-art approaches. The code for DHSNet will be available at this https URL.
Abstract (translated)
由于获取高光谱图像（HSI）的标注数据难度较大，跨场景分类已成为遥感社区广泛采用的一种方法。这种方法涉及使用源域（SD）的有标签数据和目标域（TD）的无标签数据训练模型，并在目标域进行推断。然而，同一对象在源域与目标域之间的反射光谱差异以及同一地物类别在不同域中的特征分布差异对跨场景分类的性能构成了重大挑战。为解决这一问题，我们提出了一种双分类头自训练网络（DHSNet）。该方法通过跨领域对齐类别级别的特性，确保经过训练的分类器能够准确区分目标域的不同类别的数据。 我们在跨场景高光谱图像分类领域首次引入了双分类头自训练策略。所提出的这种方法不仅缩小了不同领域的差距，还防止了错误伪标签在模型中的累积。此外，我们融入了一种新颖的中心特性注意力机制来增强模型学习跨领域不变特征的能力。 实验结果表明，在三个跨场景高光谱图像数据集上的测试显示，我们的DHSNet显著优于其他最先进的方法。有关DHSNet的代码将在以下网址提供：[请在此处插入具体URL]。
URL
https://arxiv.org/abs/2502.17879
PDF
-
Variational Bayesian Adaptive Learning of Deep Latent Variables for Acoustic Knowledge Transfer
2025-01-26 11:54:41Hu Hu, Sabato Marco Siniscalchi, Chao-Han Huck Yang, Chin-Hui LeeAbstract
In this work, we propose a novel variational Bayesian adaptive learning approach for cross-domain knowledge transfer to address acoustic mismatches between training and testing conditions, such as recording devices and environmental noise. Different from the traditional Bayesian approaches that impose uncertainties on model parameters risking the curse of dimensionality due to the huge number of parameters, we focus on estimating a manageable number of latent variables in deep neural models. Knowledge learned from a source domain is thus encoded in prior distributions of deep latent variables and optimally combined, in a Bayesian sense, with a small set of adaptation data from a target domain to approximate the corresponding posterior distributions. Two different strategies are proposed and investigated to estimate the posterior distributions: Gaussian mean-field variational inference, and empirical Bayes. These strategies address the presence or absence of parallel data in the source and target domains. Furthermore, structural relationship modeling is investigated to enhance the approximation. We evaluated our proposed approaches on two acoustic adaptation tasks: 1) device adaptation for acoustic scene classification, and 2) noise adaptation for spoken command recognition. Experimental results show that the proposed variational Bayesian adaptive learning approach can obtain good improvements on target domain data, and consistently outperforms state-of-the-art knowledge transfer methods.
Abstract (translated)
在这项工作中，我们提出了一种新颖的变分贝叶斯自适应学习方法，用于跨域知识迁移，以解决训练和测试条件之间声学不匹配的问题，如录音设备和环境噪声。与传统的贝叶斯方法不同，后者通过在模型参数上施加不确定性来应对维度灾难问题（由于大量的参数数量），我们专注于估计深层神经网络中有限数量的潜在变量。从源域学到的知识因此被编码为深度潜在变量的先验分布，并在贝叶斯意义上与目标领域的少量适应数据集相结合，以近似相应的后验分布。为了估算这些后验分布，提出了并研究了两种不同的策略：高斯均值场变分推理和经验贝叶斯方法。这两种策略分别处理源域和目标域中是否存在平行数据的问题。此外，还探讨了结构关系建模以增强近似效果。 我们在两项声学适应任务上评估了我们提出的方法：1）用于声景分类的设备自适应；2）用于语音命令识别的噪声自适应。实验结果表明，所提出的变分贝叶斯自适应学习方法在目标领域数据上可以获得显著改进，并且始终优于当前最先进的知识迁移方法。
URL
https://arxiv.org/abs/2501.15496
PDF
-
A Vision-Language Framework for Multispectral Scene Representation Using Language-Grounded Features
2025-01-17 12:12:33Enes Karanfil, Nevrez Imamoglu, Erkut Erdem, Aykut ErdemAbstract
Scene understanding in remote sensing often faces challenges in generating accurate representations for complex environments such as various land use areas or coastal regions, which may also include snow, clouds, or haze. To address this, we present a vision-language framework named Spectral LLaVA, which integrates multispectral data with vision-language alignment techniques to enhance scene representation and description. Using the BigEarthNet v2 dataset from Sentinel-2, we establish a baseline with RGB-based scene descriptions and further demonstrate substantial improvements through the incorporation of multispectral information. Our framework optimizes a lightweight linear projection layer for alignment while keeping the vision backbone of SpectralGPT frozen. Our experiments encompass scene classification using linear probing and language modeling for jointly performing scene classification and description generation. Our results highlight Spectral LLaVA's ability to produce detailed and accurate descriptions, particularly for scenarios where RGB data alone proves inadequate, while also enhancing classification performance by refining SpectralGPT features into semantically meaningful representations.
Abstract (translated)
遥感中的场景理解常常面临挑战，尤其是在生成复杂环境（如各种土地利用区域或沿海地区）的准确表示时，这些环境中可能还包括雪、云或雾霾等干扰因素。为了解决这些问题，我们提出了一种视觉-语言框架，名为Spectral LLaVA，该框架将多光谱数据与视觉-语言对齐技术相结合，以增强场景的表现和描述能力。使用Sentinel-2卫星的BigEarthNet v2数据集，我们在基于RGB图像的场景描述基础上建立了基准，并通过引入多光谱信息进一步展示了显著改进。 我们的框架优化了一层轻量级线性投影层来实现视觉与语言的对齐，同时保持SpectralGPT的视觉骨干网络不变。实验涵盖了使用线性探测进行场景分类以及联合执行场景分类和描述生成的语言建模任务。我们的结果突显了Spectral LLaVA在产生详细且准确描述方面的能力，特别是在仅依靠RGB数据不足以充分描述的情况下表现尤为突出，并通过将SpectralGPT的特征精炼为语义上有意义的表现来提高分类性能。 简而言之，Spectral LLaVA框架不仅能够生成对复杂环境场景的细致和精确描述，还能够在多光谱图像处理中提升模型在场景理解任务中的效果。
URL
https://arxiv.org/abs/2501.10144
PDF
-
LWGANet: A Lightweight Group Attention Backbone for Remote Sensing Visual Tasks
2025-01-17 08:56:17Wei Lu, Si-Bao Chen, Chris H. Q. Ding, Jin Tang, Bin LuoAbstract
Remote sensing (RS) visual tasks have gained significant academic and practical importance. However, they encounter numerous challenges that hinder effective feature extraction, including the detection and recognition of multiple objects exhibiting substantial variations in scale within a single image. While prior dual-branch or multi-branch architectural strategies have been effective in managing these object variances, they have concurrently resulted in considerable increases in computational demands and parameter counts. Consequently, these architectures are rendered less viable for deployment on resource-constrained devices. Contemporary lightweight backbone networks, designed primarily for natural images, frequently encounter difficulties in effectively extracting features from multi-scale objects, which compromises their efficacy in RS visual tasks. This article introduces LWGANet, a specialized lightweight backbone network tailored for RS visual tasks, incorporating a novel lightweight group attention (LWGA) module designed to address these specific challenges. LWGA module, tailored for RS imagery, adeptly harnesses redundant features to extract a wide range of spatial information, from local to global scales, without introducing additional complexity or computational overhead. This facilitates precise feature extraction across multiple scales within an efficient this http URL was rigorously evaluated across twelve datasets, which span four crucial RS visual tasks: scene classification, oriented object detection, semantic segmentation, and change detection. The results confirm LWGANet's widespread applicability and its ability to maintain an optimal balance between high performance and low complexity, achieving SOTA results across diverse datasets. LWGANet emerged as a novel solution for resource-limited scenarios requiring robust RS image processing capabilities.
Abstract (translated)
遥感（RS）视觉任务在学术和实际应用中具有重要意义，但它们面临着许多挑战，这些挑战阻碍了有效的特征提取，包括在同一图像内检测并识别多个尺寸变化显著的对象。尽管先前的双分支或多分支架构策略在处理对象变异方面表现出了有效性，但同时也导致了计算需求和参数数量的巨大增加。因此，在资源受限设备上部署这些架构变得不那么可行。目前为自然图像设计的轻量级骨干网络通常难以有效地从多尺度物体中提取特征，从而影响其在遥感视觉任务中的效能。本文介绍了LWGANet，这是一种专门针对遥感视觉任务定制的轻量化骨干网络，并引入了新型的轻量级组注意力（LWGA）模块来解决这些特定挑战。LWGA模块专为遥感图像设计，能够有效地利用冗余特征提取从局部到全局尺度的广泛空间信息，同时不增加复杂性或计算开销。这使得在高效框架内进行多尺度精确特征提取成为可能。该研究在涵盖四个关键RS视觉任务（场景分类、定向物体检测、语义分割和变化检测）的十二个数据集上进行了严格的评估。结果证实了LWGANet广泛的适用性和其保持高性能与低复杂性之间最佳平衡的能力，在各种数据集中取得了最先进的成果。LWGANet成为了一个新的解决方案，适用于需要强大遥感图像处理能力但资源有限的情境。
URL
https://arxiv.org/abs/2501.10040
PDF
-
Quantum-Enhanced Transformers for Robust Acoustic Scene Classification in IoT Environments
2025-01-16 09:06:10Minh K. Quan, Mayuri Wijayasundara, Sujeeva Setunge, Pubudu N. PathiranaAbstract
The proliferation of Internet of Things (IoT) devices equipped with acoustic sensors necessitates robust acoustic scene classification (ASC) capabilities, even in noisy and data-limited environments. Traditional machine learning methods often struggle to generalize effectively under such conditions. To address this, we introduce Q-ASC, a novel Quantum-Inspired Acoustic Scene Classifier that leverages the power of quantum-inspired transformers. By integrating quantum concepts like superposition and entanglement, Q-ASC achieves superior feature learning and enhanced noise resilience compared to classical models. Furthermore, we introduce a Quantum Variational Autoencoder (QVAE) based data augmentation technique to mitigate the challenge of limited labeled data in IoT deployments. Extensive evaluations on the Tampere University of Technology (TUT) Acoustic Scenes 2016 benchmark dataset demonstrate that Q-ASC achieves remarkable accuracy between 68.3% and 88.5% under challenging conditions, outperforming state-of-the-art methods by over 5% in the best case. This research paves the way for deploying intelligent acoustic sensing in IoT networks, with potential applications in smart homes, industrial monitoring, and environmental surveillance, even in adverse acoustic environments.
Abstract (translated)
物联网（IoT）设备中配备声学传感器的数量激增，要求在嘈杂且数据有限的环境中具备强大的声景分类（ASC）能力。传统机器学习方法在这种条件下往往难以有效泛化。为此，我们引入了Q-ASC，这是一种基于量子启发式变压器的新型量子启发式声景分类器，利用了诸如叠加和纠缠等量子概念的力量，从而在特征学习和抗噪性能方面超越经典模型。此外，为了缓解物联网部署中标签数据不足的问题，我们还提出了一种基于量子变分自动编码器（QVAE）的数据增强技术。 我们在坦佩雷理工大学（TUT）2016年声景基准数据集上进行了广泛的评估，结果表明，在挑战性的条件下，Q-ASC的准确性达到了惊人的68.3%到88.5%，在最佳情况下比现有方法高出超过5%。这项研究为在物联网网络中部署智能声学传感铺平了道路，并且潜在的应用包括智能家居、工业监控和环境监视，即使是在不良的声学环境中也是如此。
URL
https://arxiv.org/abs/2501.09394
PDF
-
Multi-Label Scene Classification in Remote Sensing Benefits from Image Super-Resolution
2025-01-12 05:25:16Ashitha Mudraje, Brian B. Moser, Stanislav Frolov, Andreas DengelAbstract
Satellite imagery is a cornerstone for numerous Remote Sensing (RS) applications; however, limited spatial resolution frequently hinders the precision of such systems, especially in multi-label scene classification tasks as it requires a higher level of detail and feature differentiation. In this study, we explore the efficacy of image Super-Resolution (SR) as a pre-processing step to enhance the quality of satellite images and thus improve downstream classification performance. We investigate four SR models - SRResNet, HAT, SeeSR, and RealESRGAN - and evaluate their impact on multi-label scene classification across various CNN architectures, including ResNet-50, ResNet-101, ResNet-152, and Inception-v4. Our results show that applying SR significantly improves downstream classification performance across various metrics, demonstrating its ability to preserve spatial details critical for multi-label tasks. Overall, this work offers valuable insights into the selection of SR techniques for multi-label prediction in remote sensing and presents an easy-to-integrate framework to improve existing RS systems.
Abstract (translated)
卫星图像在众多遥感（RS）应用中扮演着基石角色；然而，有限的空间分辨率常常限制了这些系统的精度，特别是在多标签场景分类任务中，因为这类任务需要更高的细节层次和特征区分度。在这项研究中，我们探索了使用图像超分辨率（SR）作为预处理步骤来提升卫星图像质量，并从而提高下游分类性能的有效性。我们评估了四种SR模型——SRResNet、HAT、SeeSR 和 RealESRGAN，在包括 ResNet-50、ResNet-101、ResNet-152 以及 Inception-v4 等不同卷积神经网络（CNN）架构上的多标签场景分类任务中的影响。我们的研究结果表明，应用超分辨率技术显著提升了各种性能指标下的下游分类效果，展示了其在保持对多标签任务至关重要的空间细节方面的能力。总的来说，这项工作为选择适用于遥感中多标签预测的超分辨率技术提供了有价值的见解，并提供了一个易于集成的框架来提升现有RS系统。
URL
https://arxiv.org/abs/2501.06720
PDF
-
Vision-Language Models for Autonomous Driving: CLIP-Based Dynamic Scene Understanding
2025-01-09 20:29:31Mohammed Elhenawy, Huthaifa I. Ashqar, Andry Rakotonirainy, Taqwa I. Alhadidi, Ahmed Jaber, Mohammad Abu TamiAbstract
Scene understanding is essential for enhancing driver safety, generating human-centric explanations for Automated Vehicle (AV) decisions, and leveraging Artificial Intelligence (AI) for retrospective driving video analysis. This study developed a dynamic scene retrieval system using Contrastive Language-Image Pretraining (CLIP) models, which can be optimized for real-time deployment on edge devices. The proposed system outperforms state-of-the-art in-context learning methods, including the zero-shot capabilities of GPT-4o, particularly in complex scenarios. By conducting frame-level analysis on the Honda Scenes Dataset, which contains a collection of about 80 hours of annotated driving videos capturing diverse real-world road and weather conditions, our study highlights the robustness of CLIP models in learning visual concepts from natural language supervision. Results also showed that fine-tuning the CLIP models, such as ViT-L/14 and ViT-B/32, significantly improved scene classification, achieving a top F1 score of 91.1%. These results demonstrate the ability of the system to deliver rapid and precise scene recognition, which can be used to meet the critical requirements of Advanced Driver Assistance Systems (ADAS). This study shows the potential of CLIP models to provide scalable and efficient frameworks for dynamic scene understanding and classification. Furthermore, this work lays the groundwork for advanced autonomous vehicle technologies by fostering a deeper understanding of driver behavior, road conditions, and safety-critical scenarios, marking a significant step toward smarter, safer, and more context-aware autonomous driving systems.
Abstract (translated)
场景理解对于提升驾驶员安全、生成针对自动驾驶汽车（AV）决策的人类可解释性，以及利用人工智能进行驾驶视频回顾分析至关重要。本研究开发了一种基于对比语言-图像预训练（CLIP）模型的动态场景检索系统，该系统可以优化为在边缘设备上的实时部署。所提出的系统在复杂场景中比最先进的情境学习方法表现更优，包括GPT-4o的零样本能力。通过对本田场景数据集进行帧级分析——这是一个包含大约80小时注释驾驶视频的数据集，涵盖了各种真实世界的道路和天气条件——我们的研究强调了CLIP模型从自然语言监督中学习视觉概念的能力之强健性。 实验结果还表明，对ViT-L/14和ViT-B/32等CLIP模型进行微调能够显著提高场景分类的准确性，达到了最高的F1分数为91.1%。这些结果展示了该系统能够提供快速且精确的场景识别能力，可用于满足高级驾驶辅助系统（ADAS）的关键需求。 这项研究展示了CLIP模型在动态场景理解和分类中具有提供可扩展和高效框架的巨大潜力。此外，本工作通过促进对驾驶员行为、道路条件及安全关键情景的理解，为先进自动驾驶技术的发展奠定了基础，标志着向更智能、更安全、更具上下文感知的自主驾驶系统迈进的重要一步。
URL
https://arxiv.org/abs/2501.05566
PDF
-
Enhancing Scene Classification in Cloudy Image Scenarios: A Collaborative Transfer Method with Information Regulation Mechanism using Optical Cloud-Covered and SAR Remote Sensing Images
2025-01-08 05:14:36Yuze Wang, Rong Xiao, Haifeng Li, Mariana Belgiu, Chao TaoAbstract
In remote sensing scene classification, leveraging the transfer methods with well-trained optical models is an efficient way to overcome label scarcity. However, cloud contamination leads to optical information loss and significant impacts on feature distribution, challenging the reliability and stability of transferred target models. Common solutions include cloud removal for optical data or directly using Synthetic aperture radar (SAR) data in the target domain. However, cloud removal requires substantial auxiliary data for support and pre-training, while directly using SAR disregards the unobstructed portions of optical data. This study presents a scene classification transfer method that synergistically combines multi-modality data, which aims to transfer the source domain model trained on cloudfree optical data to the target domain that includes both cloudy optical and SAR data at low cost. Specifically, the framework incorporates two parts: (1) the collaborative transfer strategy, based on knowledge distillation, enables the efficient prior knowledge transfer across heterogeneous data; (2) the information regulation mechanism (IRM) is proposed to address the modality imbalance issue during transfer. It employs auxiliary models to measure the contribution discrepancy of each modality, and automatically balances the information utilization of modalities during the target model learning process at the sample-level. The transfer experiments were conducted on simulated and real cloud datasets, demonstrating the superior performance of the proposed method compared to other solutions in cloud-covered scenarios. We also verified the importance and limitations of IRM, and further discussed and visualized the modality imbalance problem during the model transfer. Codes are available at this https URL
Abstract (translated)
在遥感场景分类中，利用经过良好训练的光学模型进行迁移学习是一种克服标签稀缺的有效方法。然而，云污染会导致光学信息丢失，并对特征分布产生显著影响，从而挑战了转移目标模型的可靠性和稳定性。常见的解决方案包括去除光学数据中的云层或直接使用合成孔径雷达（SAR）数据作为目标领域。但是，云层移除需要大量的辅助数据进行支持和预训练，而直接使用SAR则忽略了未被遮挡部分的光学数据。 本研究提出了一种场景分类迁移方法，该方法协同结合了多模态数据，旨在以低成本将源域中在无云条件下训练好的光学模型迁移到目标领域，其中包含有云的光学数据和SAR数据。具体而言，框架包括两个部分：（1）基于知识蒸馏的合作转移策略，能够有效地跨异构数据进行先验知识迁移；（2）信息调节机制（IRM），用于解决在迁移过程中模态不平衡的问题。该机制采用辅助模型来测量每种模态的贡献差异，并在目标模型学习过程中的样本级别自动平衡各种模态的信息利用。 转移实验是在模拟和真实云数据集上进行的，结果表明，在被云覆盖的情况下，所提出的方法相较于其他解决方案具有优越的表现。我们还验证了IRM的重要性和局限性，并进一步讨论并可视化了在模型迁移过程中出现的模态不平衡问题。代码可在以下链接获取：[此处插入实际URL]
URL
https://arxiv.org/abs/2501.04283
PDF
-
FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models
2025-01-05 07:10:27Hui Lin, Chao Zhang, Danfeng Hong, Kexin Dong, Congcong WenAbstract
Remote sensing data is often distributed across multiple institutions, and due to privacy concerns and data-sharing restrictions, leveraging large-scale datasets in a centralized training framework is challenging. Federated learning offers a promising solution by enabling collaborative model training across distributed data sources without requiring data centralization. However, current Vision-Language Models (VLMs), which typically contain billions of parameters, pose significant communication challenges for traditional federated learning approaches based on model parameter updates, as they would incur substantial communication costs. In this paper, we propose FedRSCLIP, the first federated learning framework designed for remote sensing image classification based on a VLM, specifically CLIP. FedRSCLIP addresses the challenges of data heterogeneity and large-scale model transmission in federated environments by introducing Prompt Learning, which optimizes only a small set of tunable parameters. The framework introduces a dual-prompt mechanism, comprising Shared Prompts for global knowledge sharing and Private Prompts for client-specific adaptation. To maintain semantic coherence between shared and private prompts, we propose the Dual Prompt Alignment Constraint to balance global consistency and local adaptability across diverse client distributions. Additionally, to enhance cross-modal representation learning, we introduce the Cross-Modal Feature Alignment Constraint to align multimodal features between text and image prompts. To validate the effectiveness of our proposed model, we construct a Fed-RSIC dataset based on three existing remote sensing image classification datasets, specifically designed to simulate various federated learning configurations. Experimental results demonstrate the effectiveness and superiority of FedRSCLIP in remote sensing image classification.
Abstract (translated)
遥感数据通常分布在多个机构之间，由于隐私顾虑和数据共享限制，在集中式训练框架中利用大规模的数据集具有挑战性。联邦学习通过允许多个分布式数据源协同模型训练而不需进行数据集中化提供了一个有前景的解决方案。然而，现有的视觉-语言模型（VLMs）通常包含数十亿参数，这对基于模型参数更新的传统联邦学习方法提出了显著的通信挑战，因为这将带来高昂的通信成本。在本文中，我们提出FedRSCLIP，这是首个为基于视觉-语言模型CLIP的遥感图像分类设计的联邦学习框架。FedRSCLIP通过引入提示学习解决了数据异构性和大规模模型传输在联邦环境中的挑战，仅优化一小部分可调参数。该框架引入了双提示机制，包括用于全局知识共享的共享提示和针对特定客户端适应的私有提示。为了保持共享与私人提示之间的语义一致性，我们提出了双提示对齐约束以在全球一致性和局部适应性之间平衡各种客户端分布情况下的差异。此外，为增强跨模态表示学习，我们引入了跨模态特征对齐约束，用于调整文本和图像提示之间的多模式特征。为了验证所提出模型的有效性，我们基于三个现有的遥感图像分类数据集构建了一个Fed-RSIC数据集，专门设计以模拟各种联邦学习配置。实验结果证明了FedRSCLIP在遥感图像分类中的有效性和优越性。
URL
https://arxiv.org/abs/2501.02461
PDF
-
UniRS: Unifying Multi-temporal Remote Sensing Tasks through Vision Language Models
2024-12-30 06:34:18Yujie Li, Wenjia Xu, Guangzuo Li, Zijian Yu, Zhiwei Wei, Jiuniu Wang, Mugen PengAbstract
The domain gap between remote sensing imagery and natural images has recently received widespread attention and Vision-Language Models (VLMs) have demonstrated excellent generalization performance in remote sensing multimodal tasks. However, current research is still limited in exploring how remote sensing VLMs handle different types of visual inputs. To bridge this gap, we introduce \textbf{UniRS}, the first vision-language model \textbf{uni}fying multi-temporal \textbf{r}emote \textbf{s}ensing tasks across various types of visual input. UniRS supports single images, dual-time image pairs, and videos as input, enabling comprehensive remote sensing temporal analysis within a unified framework. We adopt a unified visual representation approach, enabling the model to accept various visual inputs. For dual-time image pair tasks, we customize a change extraction module to further enhance the extraction of spatiotemporal features. Additionally, we design a prompt augmentation mechanism tailored to the model's reasoning process, utilizing the prior knowledge of the general-purpose VLM to provide clues for UniRS. To promote multi-task knowledge sharing, the model is jointly fine-tuned on a mixed dataset. Experimental results show that UniRS achieves state-of-the-art performance across diverse tasks, including visual question answering, change captioning, and video scene classification, highlighting its versatility and effectiveness in unifying these multi-temporal remote sensing tasks. Our code and dataset will be released soon.
Abstract (translated)
最近，遥感图像与自然图像之间的领域差距引起了广泛的关注，并且视觉-语言模型（VLMs）在多模态遥感任务中展示了出色的泛化性能。然而，目前的研究仍然局限于探索遥感 VLM 如何处理不同类型的数据输入。为了弥补这一不足，我们引入了 **UniRS**，这是第一个统一跨多种类型视觉输入的多时态遥感任务的视觉-语言模型。UniRS 支持单幅图像、双时间图像对和视频作为输入，在一个统一框架内实现了全面的遥感时态分析。我们采用了一种统一的视觉表示方法，使模型能够接受各种视觉输入。对于双时间图像对任务，我们定制了一个变化提取模块，以进一步增强空间-时间特征的抽取能力。此外，我们设计了一种针对模型推理过程优化的提示增强机制，利用通用 VLM 的先验知识为 UniRS 提供线索。为了促进多任务的知识共享，该模型在混合数据集上进行了联合微调。实验结果表明，UniRS 在视觉问答、变化描述和视频场景分类等多种任务中均达到了最先进的性能，突显了其在统一这些多时态遥感任务中的多功能性和有效性。我们的代码和数据集将很快发布。
URL
https://arxiv.org/abs/2412.20742
PDF
-
Improving Acoustic Scene Classification in Low-Resource Conditions
2024-12-30 05:42:20Zhi Chen, Yun-Fei Shao, Yong Ma, Mingsheng Wei, Le Zhang, Wei-Qiang ZhangAbstract
Acoustic Scene Classification (ASC) identifies an environment based on an audio signal. This paper explores ASC in low-resource conditions and proposes a novel model, DS-FlexiNet, which combines depthwise separable convolutions from MobileNetV2 with ResNet-inspired residual connections for a balance of efficiency and accuracy. To address hardware limitations and device heterogeneity, DS-FlexiNet employs Quantization Aware Training (QAT) for model compression and data augmentation methods like Auto Device Impulse Response (ADIR) and Freq-MixStyle (FMS) to improve cross-device generalization. Knowledge Distillation (KD) from twelve teacher models further enhances performance on unseen devices. The architecture includes a custom Residual Normalization layer to handle domain differences across devices, and depthwise separable convolutions reduce computational overhead without sacrificing feature representation. Experimental results show that DS-FlexiNet excels in both adaptability and performance under resource-constrained conditions.
Abstract (translated)
声学场景分类（ASC）是一种基于音频信号识别环境的技术。本文探讨了在低资源条件下进行声学场景分类，并提出了一种新型模型DS-FlexiNet，该模型结合了MobileNetV2中的深度可分离卷积和ResNet启发式的残差连接，在效率与准确性之间取得了平衡。为了应对硬件限制和设备异质性问题，DS-FlexiNet采用量化感知训练（QAT）进行模型压缩，并使用诸如自动设备脉冲响应（ADIR）和频谱混合样式增强（FMS）等数据增强方法来提高跨设备泛化能力。从十二个教师模型中通过知识蒸馏进一步提升了在未见过的设备上的性能表现。该架构还包括一个自定义的残差归一化层，用于处理不同设备之间的领域差异，并且深度可分离卷积减少了计算负担而不牺牲特征表示的能力。 实验结果表明，在资源受限条件下，DS-FlexiNet无论是在适应性还是性能方面均表现出色。
URL
https://arxiv.org/abs/2412.20722
PDF
-
How Certain are Uncertainty Estimates? Three Novel Earth Observation Datasets for Benchmarking Uncertainty Quantification in Machine Learning
2024-12-09 12:50:27Yuanyuan Wang, Qian Song, Dawood Wasif, Muhammad Shahzad, Christoph Koller, Jonathan Bamber, Xiao Xiang ZhuAbstract
Uncertainty quantification (UQ) is essential for assessing the reliability of Earth observation (EO) products. However, the extensive use of machine learning models in EO introduces an additional layer of complexity, as those models themselves are inherently uncertain. While various UQ methods do exist for machine learning models, their performance on EO datasets remains largely unevaluated. A key challenge in the community is the absence of the ground truth for uncertainty, i.e. how certain the uncertainty estimates are, apart from the labels for the image/signal. This article fills this gap by introducing three benchmark datasets specifically designed for UQ in EO machine learning models. These datasets address three common problem types in EO: regression, image segmentation, and scene classification. They enable a transparent comparison of different UQ methods for EO machine learning models. We describe the creation and characteristics of each dataset, including data sources, preprocessing steps, and label generation, with a particular focus on calculating the reference uncertainty. We also showcase baseline performance of several machine learning models on each dataset, highlighting the utility of these benchmarks for model development and comparison. Overall, this article offers a valuable resource for researchers and practitioners working in artificial intelligence for EO, promoting a more accurate and reliable quality measure of the outputs of machine learning models. The dataset and code are accessible via this https URL.
Abstract (translated)
不确定性量化（UQ）对于评估地球观测（EO）产品的可靠性至关重要。然而，在地球观测中广泛使用机器学习模型引入了额外的复杂性，因为这些模型本身具有内在的不确定性。虽然存在多种针对机器学习模型的UQ方法，但它们在EO数据集上的性能大多未经过充分评估。社区内的一大挑战是缺乏关于不确定性的地面真实值，即除了图像/信号标签之外，如何确定不确定性估计的确切程度。本文通过引入三个专门设计用于EO机器学习模型中UQ的基准数据集填补了这一空白。这些数据集针对地球观测中的三种常见问题类型：回归、图像分割和场景分类。它们使得不同UQ方法在EO机器学习模型上的透明比较成为可能。我们详细描述了每个数据集的创建过程及特征，包括数据来源、预处理步骤和标签生成，特别关注参考不确定性的计算。此外，我们也展示了多个机器学习模型在这三个数据集上的基线性能，强调了这些基准对于模型开发与对比的价值。总的来说，本文为从事地球观测领域人工智能研究的研究人员和实践者提供了一个宝贵的资源，推动了对机器学习模型输出质量更加准确可靠衡量的发展。该数据集和代码可以通过这个 https URL 访问。
URL
https://arxiv.org/abs/2412.06451
PDF
