Abstract
Self-supervised learning (SSL) has advanced speech processing. However, existing speech SSL methods typically assume a single sampling rate and struggle with mixed-rate data due to temporal resolution mismatch. To address this limitation, we propose MSRHuBERT, a multi-sampling-rate adaptive pre-training method. Building on HuBERT, we replace its single-rate downsampling CNN with a multi-sampling-rate adaptive downsampling CNN that maps raw waveforms from different sampling rates to a shared temporal resolution without resampling. This design enables unified mixed-rate pre-training and fine-tuning. In experiments spanning 16 to 48 kHz, MSRHuBERT outperforms HuBERT on speech recognition and full-band speech reconstruction, preserving high-frequency detail while modeling low-frequency semantic structure. Moreover, MSRHuBERT retains HuBERT's mask-prediction objective and Transformer encoder, so existing analyses and improvements that were developed for HuBERT can apply directly.
Abstract (translated)
自监督学习（SSL）推动了语音处理的发展。然而，现有的语音SSL方法通常假设单一采样率，并因时间分辨率不匹配而难以处理混合采样率数据。为应对这一局限，我们提出了MSRHuBERT，一种多采样率自适应预训练方法。该方法在HuBERT基础上，将其单采样率下采样CNN替换为多采样率自适应下采样CNN，可将不同采样率的原始波形映射至共享时间分辨率，无需重采样。此设计支持统一的混合采样率预训练与微调。在16至48kHz范围的实验中，MSRHuBERT在语音识别与全频带语音重建任务上均优于HuBERT，既保留高频细节，又能对低频语义结构进行建模。此外，MSRHuBERT保留了HuBERT的掩码预测目标与Transformer编码器，因此此前为HuBERT开发的分析方法与改进方案可直接适用。
URL
https://arxiv.org/abs/2603.23048
