Abstract
Adapting pre-trained text Large Language Models (LLMs) into Speech Language Models (Speech LMs) via continual pretraining on speech data is promising, but often degrades the original text capabilities. We propose Multimodal Depth Upscaling, an extension of an emerging strategy in continual LLM pre-training, where new transformer layers are inserted into a frozen text LLM and only the added layers are trained on speech data. Experiments with SmolLM2-360M and SmolLM2-1.7B on 48k hours of English Automatic Speech Recognition (ASR) data show that depth up-scaling achieves ASR comparable to full fine-tuning while causing far less text degradation than both full fine-tuning and Low-Rank Adaptation (LoRA). We further show that incorporating E-Branchformer, an architecture designed for speech recognition, as the inserted layers achieves ASR that matches or surpasses full fine-tuning on the larger model while reducing text degradation by over 75% with 60% fewer trainable parameters.
Abstract (translated)
通过持续在语音数据上预训练，将预训练文本大语言模型（LLMs）适配为语音语言模型（Speech LMs）具有前景，但通常会削弱原始文本能力。我们提出多模态深度上扩（Multimodal Depth Upscaling），这是一种新兴的持续LLM预训练策略的扩展，该方法在冻结的文本LLM中插入新的Transformer层，并仅对新增层进行语音数据训练。使用SmolLM2-360M和SmolLM2-1.7B在48k小时英语自动语音识别（ASR）数据上的实验表明，深度上扩实现了与全量微调相当的ASR性能，同时文本能力退化远低于全量微调和低秩适配（LoRA）。我们进一步证明，将专为语音识别设计的E-Branchformer架构作为插入层，在较大模型上实现了匹配或超越全量微调的ASR效果，同时文本能力退化减少超过75%，且可训练参数减少60%。
URL
https://arxiv.org/abs/2604.00489
