Abstract
Training automatic speech recognition (ASR) models increasingly relies on decentralized federated learning to ensure data privacy and accessibility, producing multiple local models that require effective merging. In hybrid ASR systems, while acoustic models can be merged using established methods, the language model (LM) for rescoring the N-best speech recognition list faces challenges due to the heterogeneity of non-neural n-gram models and neural network models. This paper proposes a heterogeneous LM optimization task and introduces a match-and-merge paradigm with two algorithms: the Genetic Match-and-Merge Algorithm (GMMA), using genetic operations to evolve and pair LMs, and the Reinforced Match-and-Merge Algorithm (RMMA), leveraging reinforcement learning for efficient convergence. Experiments on seven OpenSLR datasets show RMMA achieves the lowest average Character Error Rate and better generalization than baselines, converging up to seven times faster than GMMA, highlighting the paradigm's potential for scalable, privacy-preserving ASR systems.
Abstract (translated)
训练自动语音识别（ASR）模型越来越依赖于分散式的联邦学习，以确保数据隐私和可访问性，并生成需要有效合并的多个本地模型。在混合ASR系统中，虽然可以使用已建立的方法合并声学模型，但用于重评分N-best语音识别列表的语言模型（LM）由于非神经n-gram模型与神经网络模型之间的异质性而面临挑战。 本文提出了一项针对异构LM优化的任务，并引入了匹配和合并范式，包含两种算法：使用遗传操作进化并配对语言模型的遗传匹配与合并算法（GMMA），以及利用强化学习实现高效收敛的强化匹配与合并算法（RMMA）。在七个OpenSLR数据集上的实验表明，RMMA达到了最低的平均字符错误率，并且比基线方法有更好的泛化能力，其收敛速度是GMMA的七倍快，这突显了该范式在构建可扩展和保护隐私的ASR系统中的潜力。
URL
https://arxiv.org/abs/2603.04945
