Abstract
Uncertainty quantification (UQ) is essential for assessing the reliability of Earth observation (EO) products. However, the extensive use of machine learning models in EO introduces an additional layer of complexity, as those models themselves are inherently uncertain. While various UQ methods do exist for machine learning models, their performance on EO datasets remains largely unevaluated. A key challenge in the community is the absence of the ground truth for uncertainty, i.e. how certain the uncertainty estimates are, apart from the labels for the image/signal. This article fills this gap by introducing three benchmark datasets specifically designed for UQ in EO machine learning models. These datasets address three common problem types in EO: regression, image segmentation, and scene classification. They enable a transparent comparison of different UQ methods for EO machine learning models. We describe the creation and characteristics of each dataset, including data sources, preprocessing steps, and label generation, with a particular focus on calculating the reference uncertainty. We also showcase baseline performance of several machine learning models on each dataset, highlighting the utility of these benchmarks for model development and comparison. Overall, this article offers a valuable resource for researchers and practitioners working in artificial intelligence for EO, promoting a more accurate and reliable quality measure of the outputs of machine learning models. The dataset and code are accessible via this https URL.
Abstract (translated)
不确定性量化（UQ）对于评估地球观测（EO）产品的可靠性至关重要。然而，在地球观测中广泛使用机器学习模型引入了额外的复杂性，因为这些模型本身具有内在的不确定性。虽然存在多种针对机器学习模型的UQ方法，但它们在EO数据集上的性能大多未经过充分评估。社区内的一大挑战是缺乏关于不确定性的地面真实值，即除了图像/信号标签之外，如何确定不确定性估计的确切程度。本文通过引入三个专门设计用于EO机器学习模型中UQ的基准数据集填补了这一空白。这些数据集针对地球观测中的三种常见问题类型：回归、图像分割和场景分类。它们使得不同UQ方法在EO机器学习模型上的透明比较成为可能。我们详细描述了每个数据集的创建过程及特征，包括数据来源、预处理步骤和标签生成，特别关注参考不确定性的计算。此外，我们也展示了多个机器学习模型在这三个数据集上的基线性能，强调了这些基准对于模型开发与对比的价值。总的来说，本文为从事地球观测领域人工智能研究的研究人员和实践者提供了一个宝贵的资源，推动了对机器学习模型输出质量更加准确可靠衡量的发展。该数据集和代码可以通过这个 https URL 访问。
URL
https://arxiv.org/abs/2412.06451
