Scene_Classification
-
Hierarchical Semi-Supervised Active Learning for Remote Sensing
2025-11-22 13:25:42Wei Huang, Zhitong Xiong, Chenying Liu, Xiao Xiang ZhuAbstract
The performance of deep learning models in remote sensing (RS) strongly depends on the availability of high-quality labeled data. However, collecting large-scale annotations is costly and time-consuming, while vast amounts of unlabeled imagery remain underutilized. To address this challenge, we propose a Hierarchical Semi-Supervised Active Learning (HSSAL) framework that integrates semi-supervised learning (SSL) and a novel hierarchical active learning (HAL) in a closed iterative loop. In each iteration, SSL refines the model using both labeled data through supervised learning and unlabeled data via weak-to-strong self-training, improving feature representation and uncertainty estimation. Guided by the refined representations and uncertainty cues of unlabeled samples, HAL then conducts sample querying through a progressive clustering strategy, selecting the most informative instances that jointly satisfy the criteria of scalability, diversity, and uncertainty. This hierarchical process ensures both efficiency and representativeness in sample selection. Extensive experiments on three benchmark RS scene classification datasets, including UCM, AID, and NWPU-RESISC45, demonstrate that HSSAL consistently outperforms SSL- or AL-only baselines. Remarkably, with only 8%, 4%, and 2% labeled training data on UCM, AID, and NWPU-RESISC45, respectively, HSSAL achieves over 95% of fully-supervised accuracy, highlighting its superior label efficiency through informativeness exploitation of unlabeled data. Our code will be released at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2511.18058
PDF
-
Asymmetric Cross-Modal Knowledge Distillation: Bridging Modalities with Weak Semantic Consistency
2025-11-12 02:24:24Riling Wei, Kelu Yao, Chuanguang Yang, Jin Wang, Zhuoyan Gao, Chao LiAbstract
Cross-modal Knowledge Distillation has demonstrated promising performance on paired modalities with strong semantic connections, referred to as Symmetric Cross-modal Knowledge Distillation (SCKD). However, implementing SCKD becomes exceedingly constrained in real-world scenarios due to the limited availability of paired modalities. To this end, we investigate a general and effective knowledge learning concept under weak semantic consistency, dubbed Asymmetric Cross-modal Knowledge Distillation (ACKD), aiming to bridge modalities with limited semantic overlap. Nevertheless, the shift from strong to weak semantic consistency improves flexibility but exacerbates challenges in knowledge transmission costs, which we rigorously verified based on optimal transport theory. To mitigate the issue, we further propose a framework, namely SemBridge, integrating a Student-Friendly Matching module and a Semantic-aware Knowledge Alignment module. The former leverages self-supervised learning to acquire semantic-based knowledge and provide personalized instruction for each student sample by dynamically selecting the relevant teacher samples. The latter seeks the optimal transport path by employing Lagrangian optimization. To facilitate the research, we curate a benchmark dataset derived from two modalities, namely Multi-Spectral (MS) and asymmetric RGB images, tailored for remote sensing scene classification. Comprehensive experiments exhibit that our framework achieves state-of-the-art performance compared with 7 existing approaches on 6 different model architectures across various datasets.
Abstract (translated)
URL
https://arxiv.org/abs/2511.08901
PDF
-
SpecAware: A Spectral-Content Aware Foundation Model for Unifying Multi-Sensor Learning in Hyperspectral Remote Sensing Mapping
2025-10-31 06:28:14Renjie Ji, Xue Wang, Chao Niu, Wen Zhang, Yong Mei, Kun TanAbstract
Hyperspectral imaging (HSI) is a vital tool for fine-grained land-use and land-cover (LULC) mapping. However, the inherent heterogeneity of HSI data has long posed a major barrier to developing generalized models via joint training. Although HSI foundation models have shown promise for different downstream tasks, the existing approaches typically overlook the critical guiding role of sensor meta-attributes, and struggle with multi-sensor training, limiting their transferability. To address these challenges, we propose SpecAware, which is a novel hyperspectral spectral-content aware foundation model for unifying multi-sensor learning for HSI mapping. We also constructed the Hyper-400K dataset to facilitate this research, which is a new large-scale, high-quality benchmark dataset with over 400k image patches from diverse airborne AVIRIS sensors. The core of SpecAware is a two-step hypernetwork-driven encoding process for HSI data. Firstly, we designed a meta-content aware module to generate a unique conditional input for each HSI patch, tailored to each spectral band of every sample by fusing the sensor meta-attributes and its own image content. Secondly, we designed the HyperEmbedding module, where a sample-conditioned hypernetwork dynamically generates a pair of matrix factors for channel-wise encoding, consisting of adaptive spatial pattern extraction and latent semantic feature re-projection. Thus, SpecAware gains the ability to perceive and interpret spatial-spectral features across diverse scenes and sensors. This, in turn, allows SpecAware to adaptively process a variable number of spectral channels, establishing a unified framework for joint pre-training. Extensive experiments on six datasets demonstrate that SpecAware can learn superior feature representations, excelling in land-cover semantic segmentation classification, change detection, and scene classification.
Abstract (translated)
URL
https://arxiv.org/abs/2510.27219
PDF
-
AFM-Net: Advanced Fusing Hierarchical CNN Visual Priors with Global Sequence Modeling for Remote Sensing Image Scene Classification
2025-10-31 03:55:16Yuanhao Tang, Xuechao Zou, Zhengpei Hu, Junliang Xing, Chengkun Zhang, Jianqiang HuangAbstract
Remote sensing image scene classification remains a challenging task, primarily due to the complex spatial structures and multi-scale characteristics of ground objects. Existing approaches see CNNs excel at modeling local textures, while Transformers excel at capturing global context. However, efficiently integrating them remains a bottleneck due to the high computational cost of Transformers. To tackle this, we propose AFM-Net, a novel Advanced Hierarchical Fusing framework that achieves effective local and global co-representation through two pathways: a CNN branch for extracting hierarchical visual priors, and a Mamba branch for efficient global sequence modeling. The core innovation of AFM-Net lies in its Hierarchical Fusion Mechanism, which progressively aggregates multi-scale features from both pathways, enabling dynamic cross-level feature interaction and contextual reconstruction to produce highly discriminative representations. These fused features are then adaptively routed through a Mixture-of-Experts classifier module, which dispatches them to the most suitable experts for fine-grained scene recognition. Experiments on AID, NWPU-RESISC45, and UC Merced show that AFM-Net obtains 93.72, 95.54, and 96.92 percent accuracy, surpassing state-of-the-art methods with balanced performance and efficiency. Code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2510.27155
PDF
-
Joint Analysis of Acoustic Scenes and Sound Events Based on Semi-Supervised Training of Sound Events With Partial Labels
2025-10-29 01:23:12Keisuke ImotoAbstract
Annotating time boundaries of sound events is labor-intensive, limiting the scalability of strongly supervised learning in audio detection. To reduce annotation costs, weakly-supervised learning with only clip-level labels has been widely adopted. As an alternative, partial label learning offers a cost-effective approach, where a set of possible labels is provided instead of exact weak annotations. However, partial label learning for audio analysis remains largely unexplored. Motivated by the observation that acoustic scenes provide contextual information for constructing a set of possible sound events, we utilize acoustic scene information to construct partial labels of sound events. On the basis of this idea, in this paper, we propose a multitask learning framework that jointly performs acoustic scene classification and sound event detection with partial labels of sound events. While reducing annotation costs, weakly-supervised and partial label learning often suffer from decreased detection performance due to lacking the precise event set and their temporal annotations. To better balance between annotation cost and detection performance, we also explore a semi-supervised framework that leverages both strong and partial labels. Moreover, to refine partial labels and achieve better model training, we propose a label refinement method based on self-distillation for the proposed approach with partial labels.
Abstract (translated)
URL
https://arxiv.org/abs/2510.25075
PDF
-
Few-Shot Remote Sensing Image Scene Classification with CLIP and Prompt Learning
2025-10-28 11:39:22Ivica Dimitrovski, Vlatko Spasev, Ivan KitanovskiAbstract
Remote sensing applications increasingly rely on deep learning for scene classification. However, their performance is often constrained by the scarcity of labeled data and the high cost of annotation across diverse geographic and sensor domains. While recent vision-language models like CLIP have shown promise by learning transferable representations at scale by aligning visual and textual modalities, their direct application to remote sensing remains suboptimal due to significant domain gaps and the need for task-specific semantic adaptation. To address this critical challenge, we systematically explore prompt learning as a lightweight and efficient adaptation strategy for few-shot remote sensing image scene classification. We evaluate several representative methods, including Context Optimization, Conditional Context Optimization, Multi-modal Prompt Learning, and Prompting with Self-Regulating Constraints. These approaches reflect complementary design philosophies: from static context optimization to conditional prompts for enhanced generalization, multi-modal prompts for joint vision-language adaptation, and semantically regularized prompts for stable learning without forgetting. We benchmark these prompt-learning methods against two standard baselines: zero-shot CLIP with hand-crafted prompts and a linear probe trained on frozen CLIP features. Through extensive experiments on multiple benchmark remote sensing datasets, including cross-dataset generalization tests, we demonstrate that prompt learning consistently outperforms both baselines in few-shot scenarios. Notably, Prompting with Self-Regulating Constraints achieves the most robust cross-domain performance. Our findings underscore prompt learning as a scalable and efficient solution for bridging the domain gap in satellite and aerial imagery, providing a strong foundation for future research in this field.
Abstract (translated)
URL
https://arxiv.org/abs/2510.24321
PDF
-
DDSC: Dynamic Dual-Signal Curriculum for Data-Efficient Acoustic Scene Classification under Domain Shift
2025-10-20 09:43:29Peihong Zhang, Yuxuan Liu, Rui Sang, Zhixin Li, Yiqiang Cai, Yizhou Tan, Shengchen LiAbstract
Acoustic scene classification (ASC) suffers from device-induced domain shift, especially when labels are limited. Prior work focuses on curriculum-based training schedules that structure data presentation by ordering or reweighting training examples from easy-to-hard to facilitate learning; however, existing curricula are static, fixing the ordering or the weights before training and ignoring that example difficulty and marginal utility evolve with the learned representation. To overcome this limitation, we propose the Dynamic Dual-Signal Curriculum (DDSC), a training schedule that adapts the curriculum online by combining two signals computed each epoch: a domain-invariance signal and a learning-progress signal. A time-varying scheduler fuses these signals into per-example weights that prioritize domain-invariant examples in early epochs and progressively emphasize device-specific cases. DDSC is lightweight, architecture-agnostic, and introduces no additional inference overhead. Under the official DCASE 2024 Task~1 protocol, DDSC consistently improves cross-device performance across diverse ASC baselines and label budgets, with the largest gains on unseen-device splits.
Abstract (translated)
声学场景分类（ASC）面临着由于设备引起的领域偏移问题，尤其是在标签有限的情况下。之前的工作主要集中在基于课程的学习计划上，通过按难度从易到难排序或重新加权训练示例来结构化数据呈现以促进学习；然而，现有的课程是静态的，在训练前固定了排序或权重，并忽略了随着所学表示的变化，样本难度和边际效用也会变化。为了克服这一限制，我们提出了一种动态双信号课程（DDSC）的训练计划，该计划通过结合每轮计算出的两个信号在线调整课程：领域不变性信号和学习进度信号。一个时变调度器将这些信号融合成每个示例的权重，在早期时期优先考虑领域不变性的示例，并逐步强调设备特定的情况。 DDSC轻量级、与架构无关且不增加额外的推理开销。在官方DCASE 2024 Task 1协议下，DDSC在各种ASC基线和标签预算上持续提高了跨设备性能，在未见过设备的数据集上取得了最大的改进。
URL
https://arxiv.org/abs/2510.17345
PDF
-
Adaptive Gradient Calibration for Single-Positive Multi-Label Learning in Remote Sensing Image Scene Classification
2025-10-09 14:26:09Chenying Liu, Gianmarco Perantoni, Lorenzo Bruzzone, Xiao Xiang ZhuAbstract
Multi-label classification (MLC) offers a more comprehensive semantic understanding of Remote Sensing (RS) imagery compared to traditional single-label classification (SLC). However, obtaining complete annotations for MLC is particularly challenging due to the complexity and high cost of the labeling process. As a practical alternative, single-positive multi-label learning (SPML) has emerged, where each image is annotated with only one relevant label, and the model is expected to recover the full set of labels. While scalable, SPML introduces significant supervision ambiguity, demanding specialized solutions for model training. Although various SPML methods have been proposed in the computer vision domain, research in the RS context remains limited. To bridge this gap, we propose Adaptive Gradient Calibration (AdaGC), a novel and generalizable SPML framework tailored to RS imagery. AdaGC adopts a gradient calibration (GC) mechanism combined with Mixup and a dual exponential moving average (EMA) module for robust pseudo-label generation. To maximize AdaGC's effectiveness, we introduce a simple yet theoretically grounded indicator to adaptively trigger GC after an initial warm-up stage based on training dynamics, thereby guaranteeing the effectiveness of GC in mitigating overfitting to label noise. Extensive experiments on two benchmark RS datasets under two distinct label noise types demonstrate that AdaGC achieves state-of-the-art (SOTA) performance while maintaining strong robustness across diverse settings.
Abstract (translated)
多标签分类（MLC）相比传统的单标签分类（SLC），能够为遥感（RS）影像提供更为全面的语义理解。然而，由于标注过程复杂且成本高昂，获取完整的MLC注释尤其具有挑战性。作为一种实用替代方案，单一正例多标签学习（SPML）应运而生，在这种模式下，每张图像仅被标记为一个相关的标签，模型则需要恢复出全部的标签集。尽管这种方法可扩展性强，但其引入了显著的学习监督模糊性，并且要求专门的方法来解决模型训练中的问题。虽然计算机视觉领域已经提出了各种SPML方法，但在遥感领域的研究仍然有限。 为了填补这一空白，我们提出了一种新的、具有通用性的SPML框架——自适应梯度校准（AdaGC），该框架特别针对遥感影像进行了优化。AdaGC采用了结合了Mixup和双指数移动平均（EMA）模块的梯度校准机制，以生成鲁棒伪标签。为了最大化AdaGC的效果，我们引入了一个简单但理论基础坚实的指标，在初始预热阶段后根据训练动态自适应地触发梯度校准，从而确保梯度校准在缓解过度拟合到标签噪声方面发挥有效作用。 在两个基准遥感数据集上进行的广泛实验验证了AdaGC在两种不同类型的标注噪音下的表现：实验结果显示，AdaGC实现了最先进的性能，并且在各种设置下保持了强大的鲁棒性。
URL
https://arxiv.org/abs/2510.08269
PDF
-
Few-Shot Adaptation Benchmark for Remote Sensing Vision-Language Models
2025-10-08 15:29:48Karim El Khoury, Maxime Zanella, Christophe De Vleeschouwer, Benoit MacqAbstract
Remote Sensing Vision-Language Models (RSVLMs) have shown remarkable potential thanks to large-scale pretraining, achieving strong zero-shot performance on various tasks. However, their ability to generalize in low-data regimes, such as few-shot learning, remains insufficiently explored. In this work, we present the first structured benchmark for evaluating few-shot adaptation methods on RSVLMs. We conduct comprehensive experiments across ten remote sensing scene classification datasets, applying five widely used few-shot adaptation strategies to three state-of-the-art RSVLMs with varying backbones. Our findings reveal that models with similar zero-shot performance can exhibit markedly different behavior under few-shot adaptation, with some RSVLMs being inherently more amenable to such adaptation than others. The variability of performance and the absence of a clear winner among existing methods highlight the need for the development of more robust methods for few-shot adaptation tailored to RS. To facilitate future research, we provide a reproducible benchmarking framework and open-source code to systematically evaluate RSVLMs under few-shot conditions. The source code is publicly available on Github: this https URL
Abstract (translated)
远程感知视觉-语言模型（RSVLM）由于大规模预训练，在各种任务中展现了显著潜力，并实现了强大的零样本性能。然而，它们在低数据环境下（如少样本学习）的泛化能力仍需进一步探索。在这项工作中，我们首次提出了一个结构化的基准测试框架，用于评估针对远程感知视觉-语言模型的少样本适应方法。我们在十个不同的遥感场景分类数据集上进行了全面实验，将五种广泛使用的少样本适应策略应用于三种具有不同骨干网络的最先进的RSVLM。 我们的研究发现表明，即使零样本性能相似的模型，在进行少样本适应时也会表现出明显不同的行为，并且某些RSVLM比其他模型更适合这种适应。现有方法的表现差异性及缺乏明确的领先者强调了为远程感知开发更稳健的少样本适应方法的需求。 为了促进未来的研究，我们提供了一个可重复的基准测试框架和开源代码，以系统地评估在少样本条件下工作的RSVLM。源码可以在GitHub上公开获取：[此链接](https://github.com/remote-sensing-benchmark/few-shot-rs-adapter)
URL
https://arxiv.org/abs/2510.07135
PDF
-
A Novel Technique for Robust Training of Deep Networks With Multisource Weak Labeled Remote Sensing Data
2025-10-07 10:25:43Gianmarco Perantoni, Lorenzo BruzzoneAbstract
Deep learning has gained broad interest in remote sensing image scene classification thanks to the effectiveness of deep neural networks in extracting the semantics from complex data. However, deep networks require large amounts of training samples to obtain good generalization capabilities and are sensitive to errors in the training labels. This is a problem in remote sensing since highly reliable labels can be obtained at high costs and in limited amount. However, many sources of less reliable labeled data are available, e.g., obsolete digital maps. In order to train deep networks with larger datasets, we propose both the combination of single or multiple weak sources of labeled data with a small but reliable dataset to generate multisource labeled datasets and a novel training strategy where the reliability of each source is taken in consideration. This is done by exploiting the transition matrices describing the statistics of the errors of each source. The transition matrices are embedded into the labels and used during the training process to weigh each label according to the related source. The proposed method acts as a weighting scheme at gradient level, where each instance contributes with different weights to the optimization of different classes. The effectiveness of the proposed method is validated by experiments on different datasets. The results proved the robustness and capability of leveraging on unreliable source of labels of the proposed method.
Abstract (translated)
深度学习在遥感图像场景分类中因深度神经网络提取复杂数据语义的有效性而引起了广泛兴趣。然而，深度网络需要大量的训练样本以获得良好的泛化能力，并且对训练标签中的错误非常敏感。这在遥感领域是一个问题，因为高度可靠的标签只能在高成本下获取并且数量有限。但是，许多来源的低质量标注数据是可用的，例如过时的数字地图。为了使用更大的数据集来训练深度网络，我们提出了一种方法：将单一或多个弱源的数据与一个小但可靠的数据集结合，生成多源标签数据集，并提出一种新的训练策略，在该策略中考虑了每个来源的可靠性。通过利用描述每个来源错误统计信息的转换矩阵实现了这一点。这些转换矩阵被嵌入到标签中并在训练过程中使用以根据相关来源对每个标签进行加权。所提出的方法在梯度级别上充当加权方案，使得每个实例能够根据不同类别贡献不同的权重来进行优化。通过不同数据集上的实验验证了所提方法的有效性。结果证明了该方法利用不可靠的标签源的鲁棒性和能力。
URL
https://arxiv.org/abs/2510.05760
PDF
-
Lightweight and Generalizable Acoustic Scene Representations via Contrastive Fine-Tuning and Distillation
2025-10-04 08:20:50Kuang Yuan, Yang Gao, Xilin Li, Xinhao Mei, Syavosh Zadissa, Tarun Pruthi, Saeed Bagheri SereshkiAbstract
Acoustic scene classification (ASC) models on edge devices typically operate under fixed class assumptions, lacking the transferability needed for real-world applications that require adaptation to new or refined acoustic categories. We propose ContrastASC, which learns generalizable acoustic scene representations by structuring the embedding space to preserve semantic relationships between scenes, enabling adaptation to unseen categories without retraining. Our approach combines supervised contrastive fine-tuning of pre-trained models with contrastive representation distillation to transfer this structured knowledge to compact student models. Our evaluation shows that ContrastASC demonstrates improved few-shot adaptation to unseen categories while maintaining strong closed-set performance.
Abstract (translated)
声场景分类（ASC）模型在边缘设备上通常基于固定的类别假设运行，缺乏适应新或细化的声学类别的可转移性，而这对于需要进行此类调整的实际应用来说是必需的。我们提出了一种名为ContrastASC的方法，该方法通过构建嵌入空间来保持不同场景之间的语义关系，从而学习通用化的声场景表示，使模型能够无需重新训练就适应未见过的类别。我们的方法结合了预训练模型的监督对比微调与对比表征蒸馏技术，将这种结构化知识转移到紧凑的学生模型中。评估结果表明，ContrastASC在保持强闭集性能的同时，展示了对未见类别的良好少样本自适应能力。
URL
https://arxiv.org/abs/2510.03728
PDF
-
Attention over Scene Graphs: Indoor Scene Representations Toward CSAI Classification
2025-09-30 16:09:34Artur Barros, Carlos Caetano, Jo\~ao Macedo, Jefersson A. dos Santos, Sandra AvilaAbstract
Indoor scene classification is a critical task in computer vision, with wide-ranging applications that go from robotics to sensitive content analysis, such as child sexual abuse imagery (CSAI) classification. The problem is particularly challenging due to the intricate relationships between objects and complex spatial layouts. In this work, we propose the Attention over Scene Graphs for Sensitive Content Analysis (ASGRA), a novel framework that operates on structured graph representations instead of raw pixels. By first converting images into Scene Graphs and then employing a Graph Attention Network for inference, ASGRA directly models the interactions between a scene's components. This approach offers two key benefits: (i) inherent explainability via object and relationship identification, and (ii) privacy preservation, enabling model training without direct access to sensitive images. On Places8, we achieve 81.27% balanced accuracy, surpassing image-based methods. Real-world CSAI evaluation with law enforcement yields 74.27% balanced accuracy. Our results establish structured scene representations as a robust paradigm for indoor scene classification and CSAI classification. Code is publicly available at this https URL.
Abstract (translated)
室内场景分类是计算机视觉中的一个重要任务，其应用范围广泛，从机器人技术到敏感内容分析（如儿童性虐待图像(CSAI) 分类）都有涉及。该问题由于物体之间的复杂关系和空间布局的多样性而特别具有挑战性。在本文中，我们提出了一个新的框架——用于敏感内容分析的场景图注意力 (ASGRA)，它基于结构化的图表示而非原始像素进行操作。首先将图像转换为场景图，然后使用图注意力网络进行推理，ASGRA 直接建模场景组件之间的交互。这种方法提供了两个关键优势：(i) 通过识别对象和关系提供内在的可解释性；(ii) 保护隐私，在无需直接访问敏感图片的情况下就可以训练模型。 在 Places8 数据集上，我们达到了 81.27% 的平衡准确率，超过了基于图像的方法。与执法部门合作进行的真实世界 CSAI 测试中，我们的方法实现了 74.27% 的平衡准确率。我们的结果确立了结构化的场景表示作为一种稳健的室内场景分类和 CSAI 分类范式。代码可在以下网址公开获取：[此 URL] (请将此占位符替换为实际链接)。
URL
https://arxiv.org/abs/2509.26457
PDF
-
CSMoE: An Efficient Remote Sensing Foundation Model with Soft Mixture-of-Experts
2025-09-17 15:47:18Leonard Hackel, Tom Burgert, Beg\"um DemirAbstract
Self-supervised learning through masked autoencoders has attracted great attention for remote sensing (RS) foundation model (FM) development, enabling improved representation learning across diverse sensors and downstream tasks. However, existing RS FMs often either suffer from substantial computational complexity during both training and inference or exhibit limited representational capacity. These issues restrict their practical applicability in RS. To address this limitation, we propose an adaptation for enhancing the efficiency of RS FMs by integrating the Soft mixture-of-experts (MoE) mechanism into the FM. The integration of Soft MoEs into the FM allows modality-specific expert specialization alongside shared cross-sensor representation learning. To demonstrate the effectiveness of our adaptation, we apply it on the Cross-Sensor Masked Autoencoder (CSMAE) model, resulting in the Cross-Sensor Mixture-of-Experts (CSMoE) model. In addition, we introduce a thematic-climatic descriptor-driven sampling strategy for the construction of a representative and diverse training set to train our CSMoE model. Extensive experiments on scene classification, semantic segmentation, and content-based image retrieval demonstrate that our adaptation yields a reduction in computational requirements while maintaining or improving representational performance. Compared to state-of-the-art RS FMs, CSMoE achieves a superior trade-off between representational capacity, accuracy, and computational efficiency. On average, CSMoE achieves more than twice the computational efficiency of existing RS FMs, while maintaining competitive performance across all experiments. These results show the effectiveness of the proposed adaptation for creating computationally efficient RS FMs. The code for the model, the training set creation, and the model weights will be available at this https URL.
Abstract (translated)
通过掩码自动编码器进行自我监督学习在远程 sensing（RS）基础模型（FM）开发中引起了极大的关注，使各种传感器和下游任务中的表示学习得到改善。然而，现有的 RS FMs 经常要么在训练和推理过程中面临较大的计算复杂度问题，要么表现出有限的表征能力。这些问题限制了它们在 RS 中的实际应用性。 为了解决这一局限性，我们提出了一种通过将 Soft 混合专家（MoE）机制整合到基础模型中来提升 RS FMs 效率的方法。Soft MoEs 的集成使模态特异性专家专门化与共享的跨传感器表示学习相结合成为可能。为了展示我们改进的有效性，我们将这种方法应用于 Cross-Sensor Masked Autoencoder (CSMAE) 模型上，从而产生了 Cross-Sensor Mixture-of-Experts (CSMoE) 模型。 此外，我们还提出了一种主题气候描述符驱动的采样策略来构建一个具有代表性和多样性的训练集，以用于我们的 CSMoE 模型的训练。在场景分类、语义分割和基于内容的图像检索方面的广泛实验表明，我们的改进降低了计算需求，同时保持或提高了表示性能。 与最先进的 RS FMs 相比，CSMoE 达到了表征能力、准确性和计算效率之间更佳的权衡点。平均而言，CSMoE 达到现有 RS FMs 计算效率的大约两倍，并在所有实验中均表现出竞争性的性能。这些结果表明了所提议改进的有效性，用于创建具有高计算效率的 RS 基础模型。 该模型、训练集生成代码及模型权重将在以下网址提供：[插入链接]
URL
https://arxiv.org/abs/2509.14104
PDF
-
An Entropy-Guided Curriculum Learning Strategy for Data-Efficient Acoustic Scene Classification under Domain Shift
2025-09-14 09:01:52Peihong Zhang, Yuxuan Liu, Zhixin Li, Rui Sang, Yiqiang Cai, Yizhou Tan, Shengchen LiAbstract
Acoustic Scene Classification (ASC) faces challenges in generalizing across recording devices, particularly when labeled data is limited. The DCASE 2024 Challenge Task 1 highlights this issue by requiring models to learn from small labeled subsets recorded on a few devices. These models need to then generalize to recordings from previously unseen devices under strict complexity constraints. While techniques such as data augmentation and the use of pre-trained models are well-established for improving model generalization, optimizing the training strategy represents a complementary yet less-explored path that introduces no additional architectural complexity or inference overhead. Among various training strategies, curriculum learning offers a promising paradigm by structuring the learning process from easier to harder examples. In this work, we propose an entropy-guided curriculum learning strategy to address the domain shift problem in data-efficient ASC. Specifically, we quantify the uncertainty of device domain predictions for each training sample by computing the Shannon entropy of the device posterior probabilities estimated by an auxiliary domain classifier. Using entropy as a proxy for domain invariance, the curriculum begins with high-entropy samples and gradually incorporates low-entropy, domain-specific ones to facilitate the learning of generalizable representations. Experimental results on multiple DCASE 2024 ASC baselines demonstrate that our strategy effectively mitigates domain shift, particularly under limited labeled data conditions. Our strategy is architecture-agnostic and introduces no additional inference cost, making it easily integrable into existing ASC baselines and offering a practical solution to domain shift.
Abstract (translated)
声场景分类（ASC）在泛化到不同录音设备时面临挑战，尤其是在标记数据有限的情况下。DCASE 2024 挑战赛任务1突显了这一问题，要求模型从少数设备录制的小型标签子集中学习，并且需要这些模型在严格的复杂性约束下对来自之前未见过的设备的记录进行泛化。虽然诸如数据增强和使用预训练模型等技术已广泛应用于改进模型的泛化能力，但优化训练策略提供了一种补充而较少被探索的方法，这种方法不会引入额外的架构复杂性和推理开销。 在各种训练策略中，课程学习通过从简单的例子到更难的例子来结构化学习过程，提供了一个有前景的范例。在这项工作中，我们提出了一种基于熵引导的课程学习策略，以解决数据高效ASC中的领域偏移问题。具体来说，我们通过计算辅助领域分类器估计的设备后验概率的香农熵来量化每个训练样本在设备领域的预测不确定性。利用熵作为领域不变性的代理，该课程从高熵样本开始，并逐渐纳入低熵、特定领域的样本，以促进泛化表示的学习。 我们在多个DCASE 2024 ASC基线上进行了实验结果验证，证明我们的策略有效地缓解了域偏移问题，尤其是在标记数据有限的情况下。我们的策略是架构无关的，并且不会引入额外的推理成本，使其易于集成到现有的ASC基准中，并提供了一种实用的解决方案来解决领域偏移问题。
URL
https://arxiv.org/abs/2509.11168
PDF
-
Adaptive Knowledge Distillation using a Device-Aware Teacher for Low-Complexity Acoustic Scene Classification
2025-09-11 08:48:48Seung Gyu Jeong, Seong Eun KimAbstract
In this technical report, we describe our submission for Task 1, Low-Complexity Device-Robust Acoustic Scene Classification, of the DCASE 2025 Challenge. Our work tackles the dual challenges of strict complexity constraints and robust generalization to both seen and unseen devices, while also leveraging the new rule allowing the use of device labels at test time. Our proposed system is based on a knowledge distillation framework where an efficient CP-MobileNet student learns from a compact, specialized two-teacher ensemble. This ensemble combines a baseline PaSST teacher, trained with standard cross-entropy, and a 'generalization expert' teacher. This expert is trained using our novel Device-Aware Feature Alignment (DAFA) loss, adapted from prior work, which explicitly structures the feature space for device robustness. To capitalize on the availability of test-time device labels, the distilled student model then undergoes a final device-specific fine-tuning stage. Our proposed system achieves a final accuracy of 57.93\% on the development set, demonstrating a significant improvement over the official baseline, particularly on unseen devices.
Abstract (translated)
在这份技术报告中，我们描述了针对DCASE 2025挑战赛任务1——低复杂度设备鲁棒声学场景分类的参赛方案。我们的工作旨在应对严格的复杂性限制和对已见及未见过设备的强大泛化能力这双重挑战，并且还利用了新规则允许在测试时使用设备标签的规定。我们提出的系统基于知识蒸馏框架，其中高效的CP-MobileNet学生模型从一个紧凑的专业双教师集合中学习。这个集合结合了一个基线PaSST教师（通过标准交叉熵训练）和一个“泛化专家”教师。该专家是通过我们的新方法Device-Aware Feature Alignment (DAFA) 损失进行训练的，该损失借鉴了先前的工作，并明确地构建了设备鲁棒性的特征空间结构。为了利用测试时可用的设备标签信息，蒸馏后的学生模型随后会经历一个特定于设备的最终微调阶段。我们的系统在开发集上达到了57.93%的最终准确率，相较于官方基准线，特别是在未见过的设备上取得了显著的进步。
URL
https://arxiv.org/abs/2509.09262
PDF
-
HydroVision: Predicting Optically Active Parameters in Surface Water Using Computer Vision
2025-09-02 02:12:52Shubham Laxmikant Deshmukh, Matthew Wilchek, Feras A. BatarsehAbstract
Ongoing advancements in computer vision, particularly in pattern recognition and scene classification, have enabled new applications in environmental monitoring. Deep learning now offers non-contact methods for assessing water quality and detecting contamination, both critical for disaster response and public health protection. This work introduces HydroVision, a deep learning-based scene classification framework that estimates optically active water quality parameters including Chlorophyll-Alpha, Chlorophylls, Colored Dissolved Organic Matter (CDOM), Phycocyanins, Suspended Sediments, and Turbidity from standard Red-Green-Blue (RGB) images of surface water. HydroVision supports early detection of contamination trends and strengthens monitoring by regulatory agencies during external environmental stressors, industrial activities, and force majeure events. The model is trained on more than 500,000 seasonally varied images collected from the United States Geological Survey Hydrologic Imagery Visualization and Information System between 2022 and 2024. This approach leverages widely available RGB imagery as a scalable, cost-effective alternative to traditional multispectral and hyperspectral remote sensing. Four state-of-the-art convolutional neural networks (VGG-16, ResNet50, MobileNetV2, DenseNet121) and a Vision Transformer are evaluated through transfer learning to identify the best-performing architecture. DenseNet121 achieves the highest validation performance, with an R2 score of 0.89 in predicting CDOM, demonstrating the framework's promise for real-world water quality monitoring across diverse conditions. While the current model is optimized for well-lit imagery, future work will focus on improving robustness under low-light and obstructed scenarios to expand its operational utility.
Abstract (translated)
计算机视觉领域的持续进步，尤其是在模式识别和场景分类方面的发展，已经为环境监测带来了新的应用。深度学习现在提供了一种非接触式的方法来评估水质并检测污染，这对于灾难应对和公共卫生保护至关重要。本文介绍了HydroVision，这是一个基于深度学习的场景分类框架，可以从表面水体的标准红绿蓝（RGB）图像中估算出包括叶绿素-A、总叶绿素、溶解有机物（CDOM）、藻青素、悬浮颗粒物和浑浊度在内的光活性水质参数。HydroVision支持早期污染趋势检测，并在外部环境压力、工业活动以及不可抗力事件期间增强监管机构的监测能力。 该模型是在超过50万张由美国地质调查局水文图像可视化信息系统（2022年至2024年间收集）提供的季节性变化图像上训练出来的。这种方法利用广泛可用的RGB图像作为传统多光谱和高光谱遥感的一种可扩展且成本效益高的替代方案。通过迁移学习，评估了四种最先进的卷积神经网络（VGG-16、ResNet50、MobileNetV2、DenseNet121）和一个视觉变换器，以确定最佳的架构。其中，DenseNet121实现了最高的验证性能，在预测CDOM时获得了R²得分为0.89的成绩，这表明该框架在不同条件下进行实际水质监测方面具有前景。 尽管当前模型针对光线充足的图像进行了优化，未来的工作将重点放在改进低光和受阻情况下的鲁棒性上，以扩大其操作实用性。
URL
https://arxiv.org/abs/2509.01882
PDF
-
Scene-Aware Vectorized Memory Multi-Agent Framework with Cross-Modal Differentiated Quantization VLMs for Visually Impaired Assistance
2025-08-25 16:32:32Xiangxiang Wang, Xuanyu Wang, YiJia Luo, Yongbin Yu, Manping Fan, Jingtao Zhang, Liyong RenAbstract
This study proposes the dual technological innovation framework, including a cross-modal differ entiated quantization framework for vision-language models (VLMs) and a scene-aware vectorized memory multi-agent system for visually impaired assistance. The modular framework was developed implementing differentiated processing strategies, effectively reducing memory requirements from 38GB to 16GB while maintaining model performance. The multi-agent architecture combines scene classification, vectorized memory, and multimodal interaction, enabling persistent storage and efficient retrieval of scene memories. Through perception-memory-reasoning workflows, the system provides environmental information beyond the current view using historical memories. Experiments show the quantized 19B-parameter model only experiences a 2.05% performance drop on MMBench and maintains 63.7 accuracy on OCR-VQA (original: 64.9), outperforming smaller models with equivalent memory requirements like the Molmo-7B series. The system maintains response latency between 2.83-3.52 seconds from scene analysis to initial speech output, substantially faster than non-streaming methods. This research advances computational efficiency and assistive technology, offering visually impaired users comprehensive real-time assistance in scene perception, text recognition, and navigation.
Abstract (translated)
本研究提出了双技术创新框架，包括视觉-语言模型（VLMs）的跨模态差异化量化框架以及面向视障人士辅助的场景感知向量内存多智能体系统。该模块化框架通过实施差异化的处理策略开发而成，在保持模型性能的同时有效将内存需求从38GB降至16GB。多智能体架构结合了场景分类、向量记忆和跨模态交互，能够持久存储并高效检索场景记忆。通过感知-记忆-推理工作流程，系统利用历史记忆提供超出当前视图范围的环境信息。实验表明，量化后的190亿参数模型在MMBench上仅性能下降2.05%，而在OCR-VQA上的准确率保持为63.7%（原始准确率为64.9%），优于具有同等内存需求的小型模型如Molmo-7B系列。从场景分析到初始语音输出，系统响应延迟在2.83至3.52秒之间，远快于非流式处理方法。这项研究推进了计算效率和辅助技术的发展，为视障用户提供全面的实时场景感知、文本识别和导航帮助。
URL
https://arxiv.org/abs/2508.18177
PDF
-
Contributions to Label-Efficient Learning in Computer Vision and Remote Sensing
2025-08-21 21:31:50Minh-Tan PhamAbstract
This manuscript presents a series of my selected contributions to the topic of label-efficient learning in computer vision and remote sensing. The central focus of this research is to develop and adapt methods that can learn effectively from limited or partially annotated data, and can leverage abundant unlabeled data in real-world applications. The contributions span both methodological developments and domain-specific adaptations, in particular addressing challenges unique to Earth observation data such as multi-modality, spatial resolution variability, and scene heterogeneity. The manuscript is organized around four main axes including (1) weakly supervised learning for object discovery and detection based on anomaly-aware representations learned from large amounts of background images; (2) multi-task learning that jointly trains on multiple datasets with disjoint annotations to improve performance on object detection and semantic segmentation; (3) self-supervised and supervised contrastive learning with multimodal data to enhance scene classification in remote sensing; and (4) few-shot learning for hierarchical scene classification using both explicit and implicit modeling of class hierarchies. These contributions are supported by extensive experimental results across natural and remote sensing datasets, reflecting the outcomes of several collaborative research projects. The manuscript concludes by outlining ongoing and future research directions focused on scaling and enhancing label-efficient learning for real-world applications.
Abstract (translated)
这份手稿展示了我为计算机视觉和遥感领域中标签高效学习主题所选的一系列贡献。这项研究的核心在于开发并调整能够从有限或部分标注的数据中有效学习的方法，并且能够在实际应用中利用丰富的未标注数据。这些贡献涵盖了方法论的发展以及特定领域的适应，特别是针对地球观测数据的多模态、空间分辨率变化和场景异质性等独特挑战。手稿围绕四个主要方面组织： 1. 弱监督学习：基于从大量背景图像中学到的异常感知表示来进行对象发现与检测。 2. 多任务学习：在同一套不同标注的数据集上联合训练，以提高目标检测和语义分割的性能。 3. 自监督及监督对比学习：利用多模态数据增强遥感中的场景分类。 4. 少量样本学习：通过显式和隐式的类层级模型来进行分层场景分类。 这些贡献得到了跨自然与遥感数据集的大量实验结果的支持，反映了多个合作研究项目的成果。手稿最后概述了正在进行和未来的研究方向，重点在于扩大并提升标签高效学习在实际应用中的规模和效果。
URL
https://arxiv.org/abs/2508.15973
PDF
-
ASCMamba: Multimodal Time-Frequency Mamba for Acoustic Scene Classification
2025-08-21 14:56:58Bochao Sun, Dong Wang, Han YinAbstract
Acoustic Scene Classification (ASC) is a fundamental problem in computational audition, which seeks to classify environments based on the distinctive acoustic features. In the ASC task of the APSIPA ASC 2025 Grand Challenge, the organizers introduce a multimodal ASC task. Unlike traditional ASC systems that rely solely on audio inputs, this challenge provides additional textual information as inputs, including the location where the audio is recorded and the time of recording. In this paper, we present our proposed system for the ASC task in the APSIPA ASC 2025 Grand Challenge. Specifically, we propose a multimodal network, \textbf{ASCMamba}, which integrates audio and textual information for fine-grained acoustic scene understanding and effective multimodal ASC. The proposed ASCMamba employs a DenseEncoder to extract hierarchical spectral features from spectrograms, followed by a dual-path Mamba blocks that capture long-range temporal and frequency dependencies using Mamba-based state space models. In addition, we present a two-step pseudo-labeling mechanism to generate more reliable pseudo-labels. Results show that the proposed system outperforms all the participating teams and achieves a 6.2% improvement over the baseline. Code, model and pre-trained checkpoints are available at this https URL.
Abstract (translated)
声场景分类（ASC）是计算听觉学中的一个基本问题，旨在根据独特的声学特征对环境进行分类。在APSIPA ASC 2025大赛的ASC任务中，组织者引入了一个多模态ASC任务。与传统的仅依赖音频输入的传统ASC系统不同，此次挑战提供了额外的文字信息作为输入，包括录音的位置和时间。 本文介绍了我们为APSIPA ASC 2025大赛的ASC任务设计的系统。具体来说，我们提出了一种多模态网络——**ASCMamba**，该网络整合了音频与文本信息，用于细粒度声场景理解和有效的多模态ASC分类。提出的ASCMamba采用DenseEncoder从频谱图中提取分层频谱特征，并使用基于Mamba的双路径Mamba块捕捉长时延和频率依赖性。 此外，我们还提出了一种两步伪标签生成机制来产生更加可靠的伪标签。结果显示，我们的系统在所有参赛队伍中表现最优，并且相较于基线方法有6.2%的表现提升。代码、模型以及预训练的检查点可在以下链接获取：[此URL]（请将此处的"this https URL"替换为实际提供的网址）。
URL
https://arxiv.org/abs/2508.15632
PDF
-
Auditory Intelligence: Understanding the World Through Sound
2025-08-11 10:25:58Hyeonuk NamAbstract
Recent progress in auditory intelligence has yielded high-performing systems for sound event detection (SED), acoustic scene classification (ASC), automated audio captioning (AAC), and audio question answering (AQA). Yet these tasks remain largely constrained to surface-level recognition-capturing what happened but not why, what it implies, or how it unfolds in context. I propose a conceptual reframing of auditory intelligence as a layered, situated process that encompasses perception, reasoning, and interaction. To instantiate this view, I introduce four cognitively inspired task paradigms-ASPIRE, SODA, AUX, and AUGMENT-those structure auditory understanding across time-frequency pattern captioning, hierarchical event/scene description, causal explanation, and goal-driven interpretation, respectively. Together, these paradigms provide a roadmap toward more generalizable, explainable, and human-aligned auditory intelligence, and are intended to catalyze a broader discussion of what it means for machines to understand sound.
Abstract (translated)
近期在听觉智能领域的进展已经产生了高性能的声音事件检测（SED）、声学场景分类（ASC）、自动化音频字幕生成（AAC）和音频问答（AQA）系统。然而，这些任务仍然主要局限于表面级别的识别——捕捉发生了什么，但不涉及为什么发生、其含义或如何在具体情境中展开。我提议重新构想听觉智能为一个分层的、情景化的处理过程，涵盖感知、推理和互动。 为了实现这一观点，我提出了四个受认知启发的任务范式：ASPIRE、SODA、AUX 和 AUGMENT，分别结构化了时间-频率模式描述、层级事件/场景描述、因果解释以及目标驱动的解读。这些范式共同为迈向更具泛化性、可解释性和与人类相一致的听觉智能提供了一个路线图，并旨在激发关于机器如何理解声音更广泛的讨论。
URL
https://arxiv.org/abs/2508.07829
PDF
