Abstract
We present Ara-BEST-RQ, a family of self-supervised learning (SSL) models specifically designed for multi-dialectal Arabic speech processing. Leveraging 5,640 hours of crawled Creative Commons speech and combining it with publicly available datasets, we pre-train conformer-based BEST-RQ models up to 600M parameters. Our models are evaluated on dialect identification (DID) and automatic speech recognition (ASR) tasks, achieving state-of-the-art performance on the former while using fewer parameters than competing models. We demonstrate that family-targeted pre-training on Arabic dialects significantly improves downstream performance compared to multilingual or monolingual models trained on non-Arabic data. All models, code, and pre-processed datasets will be publicly released to support reproducibility and further research in Arabic speech technologies.
Abstract (translated)
我们提出了Ara-BEST-RQ，这是一系列专为多方言阿拉伯语语音处理设计的自监督学习（SSL）模型。通过利用5,640小时爬取的创作共用语音数据，并结合公开数据集，我们预训练了基于Conformer架构的BEST-RQ模型，参数量最高达6亿。我们的模型在方言识别（DID）和自动语音识别（ASR）任务上进行了评估，在前者任务上达到了最先进水平，且参数量少于竞争模型。我们证明，针对阿拉伯语方言进行的家族式预训练，相较于使用非阿拉伯语数据训练的多语言或单语模型，能显著提升下游任务性能。所有模型、代码及预处理数据集将公开释放，以支持阿拉伯语语音技术的可复现性与进一步研究。
URL
https://arxiv.org/abs/2603.21900
