Scene_Classification
-
FedX: Explanation-Guided Pruning for Communication-Efficient Federated Learning in Remote Sensing
2025-08-08 12:21:58Bar{\i}\c{s} B\"uy\"ukta\c{s}, Jonas Klotz, Beg\"um DemirAbstract
Federated learning (FL) enables the collaborative training of deep neural networks across decentralized data archives (i.e., clients), where each client stores data locally and only shares model updates with a central server. This makes FL a suitable learning paradigm for remote sensing (RS) image classification tasks, where data centralization may be restricted due to legal and privacy constraints. However, a key challenge in applying FL to RS tasks is the communication overhead caused by the frequent exchange of large model updates between clients and the central server. To address this issue, in this paper we propose a novel strategy (denoted as FedX) that uses explanation-guided pruning to reduce communication overhead by minimizing the size of the transmitted models without compromising performance. FedX leverages backpropagation-based explanation methods to estimate the task-specific importance of model components and prunes the least relevant ones at the central server. The resulting sparse global model is then sent to clients, substantially reducing communication overhead. We evaluate FedX on multi-label scene classification using the BigEarthNet-S2 dataset and single-label scene classification using the EuroSAT dataset. Experimental results show the success of FedX in significantly reducing the number of shared model parameters while enhancing the generalization capability of the global model, compared to both unpruned model and state-of-the-art pruning methods. The code of FedX will be available at this https URL.
Abstract (translated)
联邦学习（FL）使得分布式数据存储库（即客户端）之间可以协作训练深度神经网络，每个客户端本地存储数据，并仅与中央服务器共享模型更新。这使联邦学习成为遥感（RS）图像分类任务的合适学习范式，在这些任务中，由于法律和隐私限制的原因，可能无法集中化数据。然而，将联邦学习应用于遥感任务的关键挑战是频繁交换大型模型更新所导致的通信开销。为了解决这个问题，本文提出了一种新的策略（称为FedX），该策略利用解释引导剪枝来减少通信开销，在不影响性能的前提下最小化传输模型的大小。FedX采用基于反向传播的解释方法估计任务特定的模型组件的重要性，并在中央服务器上裁剪掉最不相关的组件。由此产生的稀疏全局模型随后被发送到客户端，显著减少了通信开销。 我们在多标签场景分类（使用BigEarthNet-S2数据集）和单标签场景分类（使用EuroSAT数据集）的任务中评估了FedX。实验结果表明，与未剪枝的模型及最先进的剪枝方法相比，FedX在大幅度减少共享模型参数数量的同时增强了全局模型的泛化能力。FedX的代码将在以下链接提供：[https URL]。
URL
https://arxiv.org/abs/2508.06256
PDF
-
Dual-Stream Global-Local Feature Collaborative Representation Network for Scene Classification of Mining Area
2025-07-31 09:00:13Shuqi Fan, Haoyi Wang, Xianju LiAbstract
Scene classification of mining areas provides accurate foundational data for geological environment monitoring and resource development planning. This study fuses multi-source data to construct a multi-modal mine land cover scene classification dataset. A significant challenge in mining area classification lies in the complex spatial layout and multi-scale characteristics. By extracting global and local features, it becomes possible to comprehensively reflect the spatial distribution, thereby enabling a more accurate capture of the holistic characteristics of mining scenes. We propose a dual-branch fusion model utilizing collaborative representation to decompose global features into a set of key semantic vectors. This model comprises three key components:(1) Multi-scale Global Transformer Branch: It leverages adjacent large-scale features to generate global channel attention features for small-scale features, effectively capturing the multi-scale feature relationships. (2) Local Enhancement Collaborative Representation Branch: It refines the attention weights by leveraging local features and reconstructed key semantic sets, ensuring that the local context and detailed characteristics of the mining area are effectively integrated. This enhances the model's sensitivity to fine-grained spatial variations. (3) Dual-Branch Deep Feature Fusion Module: It fuses the complementary features of the two branches to incorporate more scene information. This fusion strengthens the model's ability to distinguish and classify complex mining landscapes. Finally, this study employs multi-loss computation to ensure a balanced integration of the modules. The overall accuracy of this model is 83.63%, which outperforms other comparative models. Additionally, it achieves the best performance across all other evaluation metrics.
Abstract (translated)
矿业区域的场景分类为地质环境监测和资源开发规划提供了准确的基础数据。本研究融合了多源数据，构建了一个多模态矿地覆盖场景分类数据集。矿区分类的一个重要挑战在于其复杂的空间布局及多尺度特性。通过提取全局与局部特征，可以全面反映空间分布情况，从而更准确地捕捉矿山场景的整体特点。为此我们提出了一种基于协作表征的双分支融合模型，将全局特征分解为一组关键语义向量集。该模型包含三个核心组件： 1. 多尺度全局Transformer支路：利用相邻的大尺度特征生成小尺度特征的全球通道注意力特性，有效捕捉多尺度特征之间的关系。 2. 局部增强协作表征支路：通过局部特性和重构的关键语义集合来优化注意权重，确保矿区的局部背景和详细特性得到有效的整合。这增强了模型对细粒度空间变化的敏感性。 3. 双分支深层特征融合模块：融合两个分支互补的特征以包含更多场景信息。这种融合加强了模型区分并分类复杂矿山景观的能力。 最后，本研究采用多损失计算方法来确保各组件之间的平衡整合。该模型的整体准确率为83.63%，超过了其他比较模型，并且在所有其他评估指标上也取得了最佳表现。
URL
https://arxiv.org/abs/2507.20216
PDF
-
Lightweight Remote Sensing Scene Classification on Edge Devices via Knowledge Distillation and Early-exit
2025-07-28 08:36:36Yang Zhao, Shusheng Li, Xueshang FengAbstract
As the development of lightweight deep learning algorithms, various deep neural network (DNN) models have been proposed for the remote sensing scene classification (RSSC) application. However, it is still challenging for these RSSC models to achieve optimal performance among model accuracy, inference latency, and energy consumption on resource-constrained edge devices. In this paper, we propose a lightweight RSSC framework, which includes a distilled global filter network (GFNet) model and an early-exit mechanism designed for edge devices to achieve state-of-the-art performance. Specifically, we first apply frequency domain distillation on the GFNet model to reduce model size. Then we design a dynamic early-exit model tailored for DNN models on edge devices to further improve model inference efficiency. We evaluate our E3C model on three edge devices across four datasets. Extensive experimental results show that it achieves an average of 1.3x speedup on model inference and over 40% improvement on energy efficiency, while maintaining high classification accuracy.
Abstract (translated)
随着轻量级深度学习算法的发展，各种用于遥感场景分类（RSSC）应用的深层神经网络（DNN）模型被提出。然而，在资源受限的边缘设备上，这些RSSC模型在模型准确度、推断延迟和能耗之间达到最优性能仍然具有挑战性。本文中，我们提出了一种轻量级的RSSC框架，该框架包括一个蒸馏全局滤波网络（GFNet）模型以及一种为边缘设备设计的提前退出机制，以实现最先进的性能。具体来说，我们首先在GFNet模型上应用频域知识蒸馏来减小模型大小；然后我们设计了一种动态提前退出模型，专门针对边缘设备上的DNN模型，进一步提高模型推断效率。 我们在三个不同边缘设备和四个数据集上评估了我们的E3C模型。大量的实验结果表明，它在模型推断速度方面平均提高了1.3倍，并且能耗效率提高了40%以上，同时保持了较高的分类准确性。
URL
https://arxiv.org/abs/2507.20623
PDF
-
Joint Feature and Output Distillation for Low-complexity Acoustic Scene Classification
2025-07-25 07:23:41Haowen Li, Ziyi Yang, Mou Wang, Ee-Leng Tan, Junwei Yeow, Santi Peksi, Woon-Seng GanAbstract
This report presents a dual-level knowledge distillation framework with multi-teacher guidance for low-complexity acoustic scene classification (ASC) in DCASE2025 Task 1. We propose a distillation strategy that jointly transfers both soft logits and intermediate feature representations. Specifically, we pre-trained PaSST and CP-ResNet models as teacher models. Logits from teachers are averaged to generate soft targets, while one CP-ResNet is selected for feature-level distillation. This enables the compact student model (CP-Mobile) to capture both semantic distribution and structural information from teacher guidance. Experiments on the TAU Urban Acoustic Scenes 2022 Mobile dataset (development set) demonstrate that our submitted systems achieve up to 59.30\% accuracy.
Abstract (translated)
该报告提出了一种双层知识蒸馏框架，采用多教师指导，用于DCASE2025任务1中的低复杂度声学场景分类（ASC）。我们提出了一种同时传输软对数和中间特征表示的知识迁移策略。具体而言，我们将PaSST和CP-ResNet模型作为预训练的教师模型。来自各教师模型的对数通过平均生成软目标，而一个CP-ResNet用于特征级别的蒸馏。这使得紧凑的学生模型（CP-Mobile）能够从教师指导下捕获语义分布和结构信息。在TAU Urban Acoustic Scenes 2022 Mobile数据集（开发集）上的实验表明，我们提交的系统达到了最高59.30%的准确率。
URL
https://arxiv.org/abs/2507.19557
PDF
-
Improving Remote Sensing Classification using Topological Data Analysis and Convolutional Neural Networks
2025-07-14 15:22:29Aaryam SharmaAbstract
Topological data analysis (TDA) is a relatively new field that is gaining rapid adoption due to its robustness and ability to effectively describe complex datasets by quantifying geometric information. In imaging contexts, TDA typically models data as filtered cubical complexes from which we can extract discriminative features using persistence homology. Meanwhile, convolutional neural networks (CNNs) have been shown to be biased towards texture based local features. To address this limitation, we propose a TDA feature engineering pipeline and a simple method to integrate topological features with deep learning models on remote sensing classification. Our method improves the performance of a ResNet18 model on the EuroSAT dataset by 1.44% achieving 99.33% accuracy, which surpasses all previously reported single-model accuracies, including those with larger architectures, such as ResNet50 (2x larger) and XL Vision Transformers (197x larger). We additionally show that our method's accuracy is 1.82% higher than our ResNet18 baseline on the RESISC45 dataset. To our knowledge, this is the first application of TDA features in satellite scene classification with deep learning. This demonstrates that TDA features can be integrated with deep learning models, even on datasets without explicit topological structures, thereby increasing the applicability of TDA. A clean implementation of our method will be made publicly available upon publication.
Abstract (translated)
拓扑数据分析（TDA）是一个相对较新的领域，由于其鲁棒性和能够通过量化几何信息有效地描述复杂数据集的能力，正在迅速被采用。在成像背景下，TDA通常将数据建模为过滤后的立方复形，并从中提取持久同调特征以区分不同的数据特征。同时，卷积神经网络（CNN）已被证明倾向于基于局部纹理的特征。为了克服这一局限性，我们提出了一种用于遥感分类的TDA特征工程管道以及一种简单的方法来将拓扑特征与深度学习模型集成起来。 我们的方法提高了在EuroSAT数据集上ResNet18模型的表现，使其准确率达到了99.33%，比之前所有单独模型报告的准确性都高，包括架构更大的ResNet50（是其两倍大）和XL视觉变换器（几乎是其197倍）。此外，在RESISC45数据集上，我们的方法相比基线ResNet18模型提高了1.82%的准确率。据我们所知，这是首次将TDA特征应用于卫星场景分类与深度学习结合的应用中，这表明TDA特征可以被集成到深度学习模型中，即使是在没有明确拓扑结构的数据集上也能增强TDA的应用范围。 在发布时，我们的方法会提供一个干净的实现版本供公众使用。
URL
https://arxiv.org/abs/2507.10381
PDF
-
On the Effectiveness of Methods and Metrics for Explainable AI in Remote Sensing Image Scene Classification
2025-07-08 12:00:24Jonas Klotz, Tom Burgert, Beg\"um DemirAbstract
The development of explainable artificial intelligence (xAI) methods for scene classification problems has attracted great attention in remote sensing (RS). Most xAI methods and the related evaluation metrics in RS are initially developed for natural images considered in computer vision (CV), and their direct usage in RS may not be suitable. To address this issue, in this paper, we investigate the effectiveness of explanation methods and metrics in the context of RS image scene classification. In detail, we methodologically and experimentally analyze ten explanation metrics spanning five categories (faithfulness, robustness, localization, complexity, randomization), applied to five established feature attribution methods (Occlusion, LIME, GradCAM, LRP, and DeepLIFT) across three RS datasets. Our methodological analysis identifies key limitations in both explanation methods and metrics. The performance of perturbation-based methods, such as Occlusion and LIME, heavily depends on perturbation baselines and spatial characteristics of RS scenes. Gradient-based approaches like GradCAM struggle when multiple labels are present in the same image, while some relevance propagation methods (LRP) can distribute relevance disproportionately relative to the spatial extent of classes. Analogously, we find limitations in evaluation metrics. Faithfulness metrics share the same problems as perturbation-based methods. Localization metrics and complexity metrics are unreliable for classes with a large spatial extent. In contrast, robustness metrics and randomization metrics consistently exhibit greater stability. Our experimental results support these methodological findings. Based on our analysis, we provide guidelines for selecting explanation methods, metrics, and hyperparameters in the context of RS image scene classification.
Abstract (translated)
针对场景分类问题的可解释人工智能（xAI）方法在遥感（RS）领域引起了广泛关注。大多数用于遥感的xAI方法和相关评估指标最初是为计算机视觉（CV）中的自然图像设计的，直接应用于遥感中可能并不合适。为了应对这一挑战，在本文中，我们探讨了在遥感图像场景分类背景下解释方法和度量标准的有效性。具体来说，我们从理论分析和实验两个方面对五种成熟的特征归因方法（遮挡、LIME、GradCAM、LRP 和 DeepLIFT）应用于三种遥感数据集的十项解释指标进行了深入探讨，这些指标涵盖了五个类别（忠实度、鲁棒性、定位、复杂性和随机化）。我们的理论分析揭示了在解释方法和评价指标方面的关键限制。基于扰动的方法（如遮挡和 LIME），其性能严重依赖于扰动基准和遥感场景的空间特性。基于梯度的方法（例如 GradCAM）在存在多个标签时难以处理，而一些相关性传播方法（LRP）可能会不对称地分配类别的重要性相对于空间范围而言。类似地，我们还发现评估指标中也存在着局限性。忠实度度量与基于扰动的方法面临同样的问题；定位度量和复杂性度量对于具有大空间范围的类别不可靠。相比之下，鲁棒性和随机化度量始终表现出更高的稳定性。我们的实验结果支持了这些理论上的发现。 根据分析，我们为遥感图像场景分类中的解释方法、指标和超参数选择提供了指导原则。
URL
https://arxiv.org/abs/2507.05916
PDF
-
Controllable Reference-Based Real-World Remote Sensing Image Super-Resolution with Generative Diffusion Priors
2025-06-30 12:45:28Ce Wang, Wanjie SunAbstract
Super-resolution (SR) techniques can enhance the spatial resolution of remote sensing images by utilizing low-resolution (LR) images to reconstruct high-resolution (HR) images, enabling more efficient large-scale earth observation applications. While single-image super-resolution (SISR) methods have shown progress, reference-based super-resolution (RefSR) offers superior performance by incorporating historical HR images alongside current LR observations. However, existing RefSR methods struggle with real-world complexities, such as cross-sensor resolution gap and significant land cover changes, often leading to under-generation or over-reliance on reference image. To address these challenges, we propose CRefDiff, a novel controllable reference-based diffusion model for real-world remote sensing image SR. To address the under-generation problem, CRefDiff is built upon the pretrained Stable Diffusion model, leveraging its powerful generative prior to produce accurate structures and textures. To mitigate over-reliance on the reference, we introduce a dual-branch fusion mechanism that adaptively integrates both local and global information from the reference image. Moreover, this novel dual-branch design enables reference strength control during inference, enhancing interactivity and flexibility of the model. Finally, a strategy named Better Start is proposed to significantly reduce the number of denoising steps, thereby accelerating the inference process. To support further research, we introduce Real-RefRSSRD, a new real-world RefSR dataset for remote sensing images, consisting of HR NAIP and LR Sentinel-2 image pairs with diverse land cover changes and significant temporal gaps. Extensive experiments on Real-RefRSSRD show that CRefDiff achieves state-of-the-art performance across various metrics and improves downstream tasks such as scene classification and semantic segmentation.
Abstract (translated)
超分辨率（SR）技术可以通过利用低分辨率（LR）图像来重建高分辨率（HR）图像，从而提高遥感图像的空间分辨率，并使大规模地球观测应用更加高效。虽然单幅图像超分辨率（SISR）方法已经取得了一些进展，但参考基于的超分辨率（RefSR）通过结合历史上的HR图像和当前的LR观察数据提供了更优越的表现。然而，现有的RefSR方法在处理现实世界的复杂性时面临挑战，例如跨传感器分辨率差异和显著的土地覆盖变化，这些问题往往导致生成不足或过度依赖参考图像。为了解决这些难题，我们提出了CRefDiff，这是一种新型可控参考扩散模型，专门用于遥感图像的真实世界超分辨率任务。 为了应对生成不足的问题，CRefDiff基于预训练的Stable Diffusion模型构建，利用其强大的生成先验来产生准确的结构和纹理信息。为了解决过度依赖参考图像的问题，我们引入了一种双分支融合机制，该机制能够自适应地整合参考图像中的局部和全局信息。此外，这种新颖的双分支设计允许在推理过程中控制参考强度，从而增强模型的交互性和灵活性。最后，提出了名为“Better Start”的策略来大幅减少去噪步骤的数量，从而加速推理过程。 为了支持进一步的研究，我们引入了Real-RefRSSRD，这是一个新的真实世界RefSR数据集，专门用于遥感图像，它包含HR NAIP和LR Sentinel-2图像对，这些图像对具有多样化的土地覆盖变化和显著的时间间隔。在Real-RefRSSRD上的广泛实验表明，CRefDiff在各种指标上达到了最先进的性能，并且提高了下游任务（如场景分类和语义分割）的性能。
URL
https://arxiv.org/abs/2506.23801
PDF
-
Remote Sensing Large Vision-Language Model: Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modeling
2025-06-27 02:31:37Sungjune Park, Yeongyun Kim, Se Yeon Kim, Yong Man RoAbstract
Large Vision and Language Models (LVLMs) have shown strong performance across various vision-language tasks in natural image domains. However, their application to remote sensing (RS) remains underexplored due to significant domain differences in visual appearances, object scales, and semantics. These discrepancies hider the effective understanding of RS scenes, which contain rich, multi-level semantic information spanning from coarse-to-fine levels. Hence, it limits the direct adaptation of existing LVLMs to RS imagery. To address this gap, we propose a novel LVLM framework tailored for RS understanding, incorporating two core components: Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modeling. First, to align multi-level visual features, we introduce the retrieval-based Semantic Augmentation Module which enriches the visual features with relevant semantics across fine-to-coarse levels (e.g., object- and scene-level information). It is designed to retrieve relevant semantic cues from a RS semantic knowledge database, followed by aggregation of semantic cues with user query and multi-level visual features, resulting in semantically enriched representation across multiple levels. Second, for Semantic-aware Expert Modeling, we design semantic experts, where each expert is responsible for processing semantic representation at different levels separately. This enables hierarchical semantic understanding from coarse to fine levels. Evaluations across multiple RS tasks-including scene classification and VQA, etc.-demonstrate that the proposed framework achieves consistent improvements across multiple semantic levels. This highlights its capability and effectiveness in bridging the gap between general LVLMs and unique demands of RS-specific vision-language understanding.
Abstract (translated)
大型视觉和语言模型（LVLM）在自然图像领域的各种视觉-语言任务中表现出色。然而，由于遥感（RS）领域在视觉外观、对象尺度和语义方面存在显著差异，这些模型在遥感中的应用仍较少被探索。这种差距阻碍了对包含从粗到细多层次语义信息的遥感场景的有效理解，从而限制了现有LVLM直接应用于遥感图像的可能性。 为解决这一缺口，我们提出了一种针对遥感理解设计的新框架，该框架融合了两个核心组件：语义增强的多级对齐和感知语义的专家建模。首先，在进行多级视觉特征对准时，我们引入基于检索的语义增强模块，通过从细到粗（如对象级和场景级）层次上与相关语义结合来丰富视觉特征。此模块旨在从遥感语义知识数据库中提取相关的语义线索，并将这些线索与用户查询及多层视觉特征进行聚合，从而生成跨多个层面的语义丰富的表示。 其次，在感知语义专家建模方面，我们设计了专门处理不同层次上语义表示的专家模型。这使从粗到细级别的分层语义理解成为可能。通过在场景分类和VQA等多遥感任务中的评估显示，所提出的框架能够在多个语义层面实现一致性的改进，突显其连接通用LVLM与特定于遥感需求的能力及有效性。
URL
https://arxiv.org/abs/2506.21863
PDF
-
Towards Scalable and Generalizable Earth Observation Data Mining via Foundation Model Composition
2025-06-25 07:02:42Man Duc ChucAbstract
Foundation models are rapidly transforming Earth Observation data mining by enabling generalizable and scalable solutions for key tasks such as scene classification and semantic segmentation. While most efforts in the geospatial domain have focused on developing large models trained from scratch using massive Earth Observation datasets, an alternative strategy that remains underexplored is the reuse and combination of existing pretrained models. In this study, we investigate whether foundation models pretrained on remote sensing and general vision datasets can be effectively combined to improve performance across a diverse set of key Earth Observation tasks. Using the GEO-Bench benchmark, we evaluate several prominent models, including Prithvi, Hiera, and DOFA, on eleven datasets covering a range of spatial resolutions, sensor modalities, and task types. The results show that feature-level ensembling of smaller pretrained models can match or exceed the performance of much larger models, while requiring less training time and computational resources. Moreover, the study highlights the potential of applying knowledge distillation to transfer the strengths of ensembles into more compact models, offering a practical path for deploying foundation models in real-world Earth Observation applications.
Abstract (translated)
基础模型正在通过提供通用且可扩展的解决方案，如场景分类和语义分割等关键任务，迅速改变地球观测数据挖掘领域。尽管地理空间领域的大部分工作主要集中在开发大型模型上，这些模型从大规模地球观测数据集中从头开始训练，但一种尚未充分探索的方法是重用和组合现有的预训练模型。在这项研究中，我们调查了预先在遥感和通用视觉数据集上进行训练的基础模型是否可以有效地结合使用，以改善一系列重要地球观测任务的性能表现。 利用GEO-Bench基准测试，我们在包括Prithvi、Hiera和DOFA在内的几个突出模型上进行了评估，这些模型覆盖了多种空间分辨率、传感器模式及任务类型，共在11个数据集上进行验证。结果显示，较小预训练模型的功能级集成可以与较大模型匹配或超越其性能，并且所需的训练时间和计算资源更少。 此外，该研究还强调了将知识蒸馏应用于将集成的优势转移到更加紧凑的模型中的潜力，为实际部署基础模型在地球观测应用中提供了一条实用路径。
URL
https://arxiv.org/abs/2506.20174
PDF
-
Acquiring and Accumulating Knowledge from Diverse Datasets for Multi-label Driving Scene Classification
2025-06-20 16:06:53Ke Li, Chenyu Zhang, Yuxin Ding, Xianbiao Hu, Ruwen QinAbstract
Driving scene identification, which assigns multiple non-exclusive class labels to a scene, provides the contextual awareness necessary for enhancing autonomous vehicles' ability to understand, reason about, and interact with the complex driving environment. As a multi-label classification problem, it is better tackled via multitasking learning. However, directly training a multi-label classification model for driving scene identification through multitask learning presents two main challenges: acquiring a balanced, comprehensively annotated multi-label dataset and balancing learning across different tasks. This paper introduces a novel learning system that synergizes knowledge acquisition and accumulation (KAA) with consistency-based active learning (CAL) to address those challenges. KAA acquires and accumulates knowledge about scene identification from various single-label datasets via monotask learning. Subsequently, CAL effectively resolves the knowledge gap caused by the discrepancy between the marginal distributions of individual attributes and their joint distribution. An ablation study on our Driving Scene Identification (DSI) dataset demonstrates a 56.1% performance increase over the baseline model pretrained on ImageNet. Of this, KAA accounts for 31.3% of the gain, and CAL contributes 24.8%. Moreover, KAA-CAL stands out as the best performer when compared to state-of-the-art (SOTA) multi-label models on two public datasets, BDD100K and HSD, achieving this while using 85% less data. The DSI dataset and the implementation code for KAA-CAL are available at this https URL .
Abstract (translated)
驾驶场景识别，即为一个场景分配多个非互斥的类别标签，提供了增强自动驾驶车辆理解、推理和与复杂驾驶环境互动所需的情境意识。作为一个多标签分类问题，它更适合通过多任务学习来解决。然而，直接通过多任务学习训练用于驾驶场景识别的多标签分类模型面临两大挑战：获取平衡且全面标注的多标签数据集以及在不同任务之间保持学习平衡。 本文介绍了一种新型的学习系统，该系统将知识获取与积累（KAA）和基于一致性的主动学习（CAL）相结合，以解决上述挑战。通过单标签数据集进行单调任务学习，KAA 从多种来源获取并积累了关于场景识别的知识。随后，CAL 有效地解决了由个体属性的边际分布与其联合分布之间的差异所导致的知识差距。 在我们构建的驾驶场景识别 (DSI) 数据集上的消融研究表明，在预训练于 ImageNet 的基础模型上实现了 56.1% 的性能提升。其中，KAA 贡献了 31.3%，CAL 则贡献了 24.8% 的性能增长。此外，当与两个公开数据集 BDD100K 和 HSD 上的最先进（SOTA）多标签模型进行比较时，KAA-CAL 在使用比这些 SOTA 模型少 85% 数据的情况下表现最佳。 DSI 数据集和 KAA-CAL 的实现代码可在此链接获取：[https URL]。
URL
https://arxiv.org/abs/2506.17101
PDF
-
MSSDF: Modality-Shared Self-supervised Distillation for High-Resolution Multi-modal Remote Sensing Image Learning
2025-06-11 02:01:36Tong Wang, Guanzhou Chen, Xiaodong Zhang, Chenxi Liu, Jiaqi Wang, Xiaoliang Tan, Wenchao Guo, Qingyuan Yang, Kaiqi ZhangAbstract
Remote sensing image interpretation plays a critical role in environmental monitoring, urban planning, and disaster assessment. However, acquiring high-quality labeled data is often costly and time-consuming. To address this challenge, we proposes a multi-modal self-supervised learning framework that leverages high-resolution RGB images, multi-spectral data, and digital surface models (DSM) for pre-training. By designing an information-aware adaptive masking strategy, cross-modal masking mechanism, and multi-task self-supervised objectives, the framework effectively captures both the correlations across different modalities and the unique feature structures within each modality. We evaluated the proposed method on multiple downstream tasks, covering typical remote sensing applications such as scene classification, semantic segmentation, change detection, object detection, and depth estimation. Experiments are conducted on 15 remote sensing datasets, encompassing 26 tasks. The results demonstrate that the proposed method outperforms existing pretraining approaches in most tasks. Specifically, on the Potsdam and Vaihingen semantic segmentation tasks, our method achieved mIoU scores of 78.30\% and 76.50\%, with only 50\% train-set. For the US3D depth estimation task, the RMSE error is reduced to 0.182, and for the binary change detection task in SECOND dataset, our method achieved mIoU scores of 47.51\%, surpassing the second CS-MAE by 3 percentage points. Our pretrain code, checkpoints, and HR-Pairs dataset can be found in this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2506.09327
PDF
-
A Challenge to Build Neuro-Symbolic Video Agents
2025-05-20 02:53:21Sahil Shah, Harsh Goel, Sai Shankar Narasimhan, Minkyu Choi, S P Sharan, Oguzhan Akcin, Sandeep ChinchaliAbstract
Modern video understanding systems excel at tasks such as scene classification, object detection, and short video retrieval. However, as video analysis becomes increasingly central to real-world applications, there is a growing need for proactive video agents for the systems that not only interpret video streams but also reason about events and take informed actions. A key obstacle in this direction is temporal reasoning: while deep learning models have made remarkable progress in recognizing patterns within individual frames or short clips, they struggle to understand the sequencing and dependencies of events over time, which is critical for action-driven decision-making. Addressing this limitation demands moving beyond conventional deep learning approaches. We posit that tackling this challenge requires a neuro-symbolic perspective, where video queries are decomposed into atomic events, structured into coherent sequences, and validated against temporal constraints. Such an approach can enhance interpretability, enable structured reasoning, and provide stronger guarantees on system behavior, all key properties for advancing trustworthy video agents. To this end, we present a grand challenge to the research community: developing the next generation of intelligent video agents that integrate three core capabilities: (1) autonomous video search and analysis, (2) seamless real-world interaction, and (3) advanced content generation. By addressing these pillars, we can transition from passive perception to intelligent video agents that reason, predict, and act, pushing the boundaries of video understanding.
Abstract (translated)
现代视频理解系统在场景分类、目标检测和短片检索等任务上表现出色。然而，随着视频分析在现实世界应用中的重要性日益增加，对于能够不仅解读视频流还能够对事件进行推理并采取明智行动的主动型视频代理的需求也在增长。这一方向上的一个关键障碍是时间推理：虽然深度学习模型在识别单帧或短片段内的模式方面取得了显著进展，但它们难以理解事件随时间排列和依赖关系，这对于基于行为的决策至关重要。解决这个限制需要超越传统的深度学习方法。我们认为，应对这一挑战需要采用神经符号学视角，即将视频查询分解为原子事件、结构化成连贯序列，并根据时间约束进行验证。这种做法可以增强可解释性，支持结构化的推理过程，并提供更有力的行为保证，这些都是构建值得信赖的视频代理的关键属性。 为此，我们向研究界提出了一个重大挑战：开发下一代智能视频代理，整合三个核心能力：(1) 自主视频搜索和分析；(2) 无缝的现实世界交互；以及 (3) 高级内容生成。通过解决这些支柱问题，我们可以从被动感知过渡到能够推理、预测并采取行动的智能视频代理，推动视频理解技术的发展边界。
URL
https://arxiv.org/abs/2505.13851
PDF
-
EarthSynth: Generating Informative Earth Observation with Diffusion Models
2025-05-17 18:27:15Jiancheng Pan, Shiye Lei, Yuqian Fu, Jiahao Li, Yanxing Liu, Yuze Sun, Xiao He, Long Peng, Xiaomeng Huang, Bo ZhaoAbstract
Remote sensing image (RSI) interpretation typically faces challenges due to the scarcity of labeled data, which limits the performance of RSI interpretation tasks. To tackle this challenge, we propose EarthSynth, a diffusion-based generative foundation model that enables synthesizing multi-category, cross-satellite labeled Earth observation for downstream RSI interpretation tasks. To the best of our knowledge, EarthSynth is the first to explore multi-task generation for remote sensing. EarthSynth, trained on the EarthSynth-180K dataset, employs the Counterfactual Composition training strategy to improve training data diversity and enhance category control. Furthermore, a rule-based method of R-Filter is proposed to filter more informative synthetic data for downstream tasks. We evaluate our EarthSynth on scene classification, object detection, and semantic segmentation in open-world scenarios, offering a practical solution for advancing RSI interpretation.
Abstract (translated)
遥感图像（RSI）解释通常面临由于标注数据稀缺而带来的挑战，这限制了RSI解释任务的性能。为了应对这一挑战，我们提出了EarthSynth，这是一种基于扩散模型的生成式基础模型，能够为下游RSI解释任务合成多类别、跨卫星平台的地球观测数据标签。据我们所知，EarthSynth是第一个探索遥感中的多任务生成的研究。 EarthSynth在EarthSynth-180K数据集上进行训练，并采用反事实组合（Counterfactual Composition）的训练策略来提高训练数据多样性并增强类别控制能力。此外，还提出了一种基于规则的方法R-Filter，用于过滤出对下游任务更有信息量的人工合成数据。 我们在开放世界的场景分类、目标检测和语义分割中评估了我们的EarthSynth模型，并为推进RSI解释提供了切实可行的解决方案。
URL
https://arxiv.org/abs/2505.12108
PDF
-
Minimizing Risk Through Minimizing Model-Data Interaction: A Protocol For Relying on Proxy Tasks When Designing Child Sexual Abuse Imagery Detection Models
2025-05-10 12:10:55Thamiris Coelho, Leo S. F. Ribeiro, Jo\~ao Macedo, Jefersson A. dos Santos, Sandra AvilaAbstract
The distribution of child sexual abuse imagery (CSAI) is an ever-growing concern of our modern world; children who suffered from this heinous crime are revictimized, and the growing amount of illegal imagery distributed overwhelms law enforcement agents (LEAs) with the manual labor of categorization. To ease this burden researchers have explored methods for automating data triage and detection of CSAI, but the sensitive nature of the data imposes restricted access and minimal interaction between real data and learning algorithms, avoiding leaks at all costs. In observing how these restrictions have shaped the literature we formalize a definition of "Proxy Tasks", i.e., the substitute tasks used for training models for CSAI without making use of CSA data. Under this new terminology we review current literature and present a protocol for making conscious use of Proxy Tasks together with consistent input from LEAs to design better automation in this field. Finally, we apply this protocol to study -- for the first time -- the task of Few-shot Indoor Scene Classification on CSAI, showing a final model that achieves promising results on a real-world CSAI dataset whilst having no weights actually trained on sensitive data.
Abstract (translated)
儿童色情虐待图像（CSAI）的传播是现代世界日益增长的关注点；遭受这种罪行侵害的孩子们再次受到伤害，而不断增加的非法图像数量则令执法人员不堪重负，不得不进行大量手动分类工作。为减轻这一负担，研究人员探索了自动化数据筛选和检测CSAI的方法，但由于这类数据的高度敏感性，实际数据与学习算法之间的接触被严格限制，以避免任何可能的数据泄露风险。观察这些限制如何影响相关文献的发展后，我们正式提出了“替代任务”（Proxy Tasks）的概念，即在不使用CSA数据的情况下训练模型的替代方法。在此新术语下，我们将回顾现有文献，并提出一种新的协议，在充分考虑执法人员的意见的同时，有意识地利用替代任务来设计更好的自动化工具。最后，我们将这一协议应用于首次研究的任务：基于CSAI的少量样本室内场景分类，展示了一个在现实世界的CSAI数据集上表现良好、但未使用敏感数据进行训练的最终模型。
URL
https://arxiv.org/abs/2505.06621
PDF
-
Low-Complexity Acoustic Scene Classification with Device Information in the DCASE 2025 Challenge
2025-05-03 08:52:18Florian Schmid, Paul Primus, Toni Heittola, Annamaria Mesaros, Irene Mart\'in-Morat\'o, Gerhard WidmerAbstract
This paper presents the Low-Complexity Acoustic Scene Classification with Device Information Task of the DCASE 2025 Challenge and its baseline system. Continuing the focus on low-complexity models, data efficiency, and device mismatch from previous editions (2022--2024), this year's task introduces a key change: recording device information is now provided at inference time. This enables the development of device-specific models that leverage device characteristics -- reflecting real-world deployment scenarios in which a model is designed with awareness of the underlying hardware. The training set matches the 25% subset used in the corresponding DCASE 2024 challenge, with no restrictions on external data use, highlighting transfer learning as a central topic. The baseline achieves 50.72% accuracy on this ten-class problem with a device-general model, improving to 51.89% when using the available device information.
Abstract (translated)
本文介绍了DCASE 2025挑战赛中的低复杂度声景分类与设备信息任务及其基准系统。延续了前几届（2022-2024年）对低复杂性模型、数据效率和设备差异化的关注，今年的任务引入了一个关键变化：在推理时提供录音设备的信息。这使得开发具有特定设备特性的模型成为可能，这些特性反映了实际部署场景中，在设计模型时会考虑到底层硬件的情况。训练集匹配了对应DCASE 2024挑战赛使用的25%子集，并且不限制使用外部数据的范围，突出了迁移学习作为主要研究课题的重要性。基准系统在不考虑设备信息的情况下对于这个十类问题达到了50.72%的准确率，在利用可用设备信息时这一数字提升到了51.89%。
URL
https://arxiv.org/abs/2505.01747
PDF
-
Streetscape Analysis with Generative AI : Vision-Language Assessment and Mapping of Urban Scenes
2025-04-23 09:08:06Joan Perez (Urban Geo Analytics, France), Giovanni Fusco (Universite Cote-Azur-CNRS-AMU-Avignon Universite, ESPACE, France)Abstract
Streetscapes are an essential component of urban space. Their assessment is presently either limited to morphometric properties of their mass skeleton or requires labor-intensive qualitative evaluations of visually perceived qualities. This paper introduces SAGAI: Streetscape Analysis with Generative Artificial Intelligence, a modular workflow for scoring street-level urban scenes using open-access data and vision-language models. SAGAI integrates OpenStreetMap geometries, Google Street View imagery, and a lightweight version of the LLaVA model to generate structured spatial indicators from images via customizable natural language prompts. The pipeline includes an automated mapping module that aggregates visual scores at both the point and street levels, enabling direct cartographic interpretation. It operates without task-specific training or proprietary software dependencies, supporting scalable and interpretable analysis of urban environments. Two exploratory case studies in Nice and Vienna illustrate SAGAI's capacity to produce geospatial outputs from vision-language inference. The initial results show strong performance for binary urban-rural scene classification, moderate precision in commercial feature detection, and lower estimates, but still informative, of sidewalk width. Fully deployable by any user, SAGAI can be easily adapted to a wide range of urban research themes, such as walkability, safety, or urban design, through prompt modification alone.
Abstract (translated)
街道景观是城市空间的重要组成部分。目前，对它们的评估要么局限于其大规模骨架的形态计量属性，要么需要进行费时费力的定性评价来衡量视觉感知的质量。本文介绍了SAGAI：一种使用开源数据和视觉语言模型进行街景分析的工作流程，旨在利用生成式人工智能为街道级别的城市场景评分。SAGAI集成了OpenStreetMap几何图形、Google Street View影像以及轻量级LLaVA模型的版本，通过可定制的自然语言提示从图像中生成结构化的空间指标。该工作流包括一个自动映射模块，可以分别在点和街区级别上聚合视觉得分，从而直接进行地图解释。SAGAI无需特定任务训练或专有软件依赖，支持城市环境的大规模且易于理解的分析。 通过尼斯和维也纳两个探索性案例研究，展示了SAGAI如何从视觉-语言推理中产生地理空间输出的能力。初步结果显示，在二元城市/农村场景分类方面表现强劲，商业特征检测方面的精度适中，人行道宽度估计较低但仍具信息价值。任何用户都可以完全部署SAGAI，并通过修改提示来轻松适应各种研究主题，如可步行性、安全性或城市设计等。
URL
https://arxiv.org/abs/2504.16538
PDF
-
FrogDogNet: Fourier frequency Retained visual prompt Output Guidance for Domain Generalization of CLIP in Remote Sensing
2025-04-23 05:35:59Hariseetharam Gunduboina (Indian Institute of Technology Bombay, India), Muhammad Haris Khan (Mohamed Bin Zayed University of Artificial Intelligence, UAE), Biplab Banerjee (Indian Institute of Technology Bombay, India)Abstract
In recent years, large-scale vision-language models (VLMs) like CLIP have gained attention for their zero-shot inference using instructional text prompts. While these models excel in general computer vision, their potential for domain generalization in remote sensing (RS) remains underexplored. Existing approaches enhance prompt learning by generating visual prompt tokens but rely on full-image features, introducing noise and background artifacts that vary within a class, causing misclassification. To address this, we propose FrogDogNet, a novel prompt learning framework integrating Fourier frequency filtering and self-attention to improve RS scene classification and domain generalization. FrogDogNet selectively retains invariant low-frequency components while eliminating noise and irrelevant backgrounds, ensuring robust feature representation across domains. The model first extracts significant features via projection and self-attention, then applies frequency-based filtering to preserve essential structural information for prompt learning. Extensive experiments on four RS datasets and three domain generalization tasks show that FrogDogNet consistently outperforms state-of-the-art prompt learning methods, demonstrating superior adaptability across domain shifts. Our findings highlight the effectiveness of frequency-based invariant feature retention in generalization, paving the way for broader applications. Our code is available at this https URL
Abstract (translated)
近年来，大型视觉-语言模型（VLM），如CLIP，因其能够使用指令文本提示进行零样本推理而备受关注。尽管这些模型在通用计算机视觉方面表现出色，但它们在遥感（RS）领域的泛化潜力却尚未被充分探索。现有的方法通过生成视觉提示令牌来增强提示学习，但是依赖于完整图像特征，这会引入噪声和背景杂质，在同一类别内变化导致误分类问题。为了解决这一问题，我们提出了FrogDogNet，这是一种新颖的提示学习框架，它结合了傅里叶频率过滤和自注意力机制，以改善RS场景分类和领域泛化能力。FrogDogNet选择性地保留不变的低频成分，并消除噪声以及不相关的背景部分，在不同领域之间确保稳健的功能表示。该模型首先通过投影和自我注意提取重要特征，然后应用基于频率的过滤来保存提示学习所需的结构信息的关键细节。 在四个RS数据集和三项域泛化任务上的大量实验表明，FrogDogNet始终优于最先进的提示学习方法，并展示了跨领域转变的强大适应性。我们的研究结果强调了频基不变特性保留在推广中的有效性，为更广泛的应用铺平道路。我们的代码可在提供的链接获取。 （注意：原文中的"this https URL"应是一个具体的网页地址，这里未提供具体网址，因此只给出一般的说明信息。）
URL
https://arxiv.org/abs/2504.16433
PDF
-
FlexiMo: A Flexible Remote Sensing Foundation Model
2025-03-31 08:46:05Xuyang Li, Chenyu Li, Pedram Ghamisi, Danfeng HongAbstract
The rapid expansion of multi-source satellite imagery drives innovation in Earth observation, opening unprecedented opportunities for Remote Sensing Foundation Models to harness diverse data. However, many existing models remain constrained by fixed spatial resolutions and patch sizes, limiting their ability to fully exploit the heterogeneous spatial characteristics inherent in satellite imagery. To address these challenges, we propose FlexiMo, a flexible remote sensing foundation model that endows the pre-trained model with the flexibility to adapt to arbitrary spatial resolutions. Central to FlexiMo is a spatial resolution-aware module that employs a parameter-free alignment embedding mechanism to dynamically recalibrate patch embeddings based on the input image's resolution and dimensions. This design not only preserves critical token characteristics and ensures multi-scale feature fidelity but also enables efficient feature extraction without requiring modifications to the underlying network architecture. In addition, FlexiMo incorporates a lightweight channel adaptation module that leverages prior spectral information from sensors. This mechanism allows the model to process images with varying numbers of channels while maintaining the data's intrinsic physical properties. Extensive experiments on diverse multimodal, multi-resolution, and multi-scale datasets demonstrate that FlexiMo significantly enhances model generalization and robustness. In particular, our method achieves outstanding performance across a range of downstream tasks, including scene classification, land cover classification, urban building segmentation, and cloud detection. By enabling parameter-efficient and physically consistent adaptation, FlexiMo paves the way for more adaptable and effective foundation models in real-world remote sensing applications.
Abstract (translated)
多源卫星图像的快速扩展推动了地球观测领域的创新，为遥感基础模型利用多样化数据提供了前所未有的机遇。然而，许多现有的模型仍然受到固定空间分辨率和补丁大小的限制，这限制了它们充分利用卫星图像中固有的异构空间特征的能力。为了解决这些挑战，我们提出了FlexiMo，这是一种灵活的遥感基础模型，它赋予预训练模型适应任意空间分辨率的灵活性。FlexiMo的核心是一个基于空间分辨率感知模块，该模块采用了一种无参数对齐嵌入机制，根据输入图像的分辨率和尺寸动态校准补丁嵌入。这种设计不仅保留了关键标记特性，并确保了多尺度特征的一致性，而且还能够在不修改底层网络架构的情况下实现高效的特征提取。 此外，FlexiMo集成了一个轻量级通道适应模块，利用传感器中的先验光谱信息。这一机制使模型能够处理不同通道数量的图像，同时保持数据的基本物理属性不变。在多种多模态、多分辨率和多尺度的数据集上进行广泛的实验表明，FlexiMo显著增强了模型的一般性和鲁棒性。特别是在场景分类、土地覆盖分类、城市建筑物分割以及云检测等一系列下游任务中，我们的方法取得了出色的性能。 通过实现参数效率高且物理属性一致的适应能力，FlexiMo为在现实世界的遥感应用中的更灵活和有效的基础模型的发展铺平了道路。
URL
https://arxiv.org/abs/2503.23844
PDF
-
Open-Vocabulary Semantic Segmentation with Uncertainty Alignment for Robotic Scene Understanding in Indoor Building Environments
2025-03-29 14:46:45Yifan Xu, Vineet Kamat, Carol MenassaAbstract
The global rise in the number of people with physical disabilities, in part due to improvements in post-trauma survivorship and longevity, has amplified the demand for advanced assistive technologies to improve mobility and independence. Autonomous assistive robots, such as smart wheelchairs, require robust capabilities in spatial segmentation and semantic recognition to navigate complex built environments effectively. Place segmentation involves delineating spatial regions like rooms or functional areas, while semantic recognition assigns semantic labels to these regions, enabling accurate localization to user-specific needs. Existing approaches often utilize deep learning; however, these close-vocabulary detection systems struggle to interpret intuitive and casual human instructions. Additionally, most existing methods ignore the uncertainty of the scene recognition problem, leading to low success rates, particularly in ambiguous and complex environments. To address these challenges, we propose an open-vocabulary scene semantic segmentation and detection pipeline leveraging Vision Language Models (VLMs) and Large Language Models (LLMs). Our approach follows a 'Segment Detect Select' framework for open-vocabulary scene classification, enabling adaptive and intuitive navigation for assistive robots in built environments.
Abstract (translated)
全球因创伤幸存率和寿命的提高，导致身体残疾人数增加，这加大了对能够改善移动能力和独立性的先进辅助技术的需求。自主辅助机器人（如智能轮椅）需要具备强大的空间分割与语义识别能力才能有效地在复杂的建筑环境中导航。空间分割涉及划分诸如房间或功能区域等空间区域，而语义识别则为这些区域分配语义标签，从而能够根据用户的特定需求进行精准定位。现有方法通常利用深度学习技术；然而，这些封闭词汇表检测系统难以理解直观且随意的人类指令。此外，大多数现有的方法忽视了场景识别问题中的不确定性，导致成功率较低，尤其是在模糊和复杂的环境中。 为了解决这些问题，我们提出了一种基于视觉语言模型（VLMs）和大规模语言模型（LLMs）的开放词汇表场景语义分割与检测流水线。我们的方法遵循“Segment Detect Select”框架进行开放词汇表场景分类，使辅助机器人能够在建筑环境中实现适应性和直观导航。
URL
https://arxiv.org/abs/2503.23105
PDF
-
Vanishing Depth: A Depth Adapter with Positional Depth Encoding for Generalized Image Encoders
2025-03-25 15:19:48Paul Koch, J\"org Kr\"uger, Ankit Chowdhury, Oliver HeimannAbstract
Generalized metric depth understanding is critical for precise vision-guided robotics, which current state-of-the-art (SOTA) vision-encoders do not support. To address this, we propose Vanishing Depth, a self-supervised training approach that extends pretrained RGB encoders to incorporate and align metric depth into their feature embeddings. Based on our novel positional depth encoding, we enable stable depth density and depth distribution invariant feature extraction. We achieve performance improvements and SOTA results across a spectrum of relevant RGBD downstream tasks - without the necessity of finetuning the encoder. Most notably, we achieve 56.05 mIoU on SUN-RGBD segmentation, 88.3 RMSE on Void's depth completion, and 83.8 Top 1 accuracy on NYUv2 scene classification. In 6D-object pose estimation, we outperform our predecessors of DinoV2, EVA-02, and Omnivore and achieve SOTA results for non-finetuned encoders in several related RGBD downstream tasks.
Abstract (translated)
通用度量深度理解对于精确的视觉引导机器人技术至关重要，而当前最先进的（SOTA）视觉编码器无法支持这一点。为了解决这个问题，我们提出了Vanishing Depth，这是一种自监督训练方法，它扩展了预训练的RGB编码器，以便将度量深度整合到其特征嵌入中，并进行对齐处理。基于我们的新型位置深度编码技术，我们能够实现稳定且与深度密度和分布无关的特征提取能力。我们在一系列相关的RGBD下游任务上实现了性能改进和SOTA结果——并且不需要微调编码器。最值得注意的是，在SUN-RGBD分割任务中我们达到了56.05 mIoU，在Void的深度完成任务中我们达到了88.3 RMSE，在NYUv2场景分类任务中我们达到了83.8 Top 1准确率。在6D物体姿态估计方面，我们超越了DinoV2、EVA-02和Omnivore等前人，并且对于几种相关的RGBD下游任务实现了未经微调编码器的SOTA结果。
URL
https://arxiv.org/abs/2503.19947
PDF
