Abstract
Large language models (LLMs) provide strong semantic priors that can improve multi-talker automatic speech recognition (MT-ASR), but using an LLM as an autoregressive decoder is computationally expensive and remains fragile under heavy overlap. In this paper, we propose an encoder-only MT-ASR framework that adapts an LLM to multi-talker conditioning and distills its semantic guidance into the encoder during training, while retaining fast CTC-style decoding at inference. Our model employs a post-encoder separator with serialized CTC to produce talker-ordered transcripts, and leverages an adapted LLM-based SOT objective as a multi-talker-aware teacher signal to explicitly regularize mixed-speech representations. To further support variable numbers of talkers, we introduce a Talker-Count Head that predicts the talker count and dynamically selects the appropriate decoding branch. Experiments on LibriMix show that the proposed encoder-only model achieves comparable performance to LLM-based systems in the two-talker condition, while delivering significant improvements in the three-talker condition with significant small RTF.
Abstract (translated)
大型语言模型（LLM）提供了强大的语义先验，可以改善多说话人自动语音识别（MT-ASR），但将LLM用作自回归解码器在计算上是昂贵的，并且在重叠严重时仍然不稳定。本文提出了一种编码器专用的多说话人语音识别框架，该框架使大型语言模型适应于多说话人的条件，并在训练过程中将其语义指导蒸馏到编码器中，同时保持快速的CTC风格解码进行推理。我们的模型采用了后编码分割器和序列化的CTC来生成有序说话人记录，并利用了基于改编LLM的SOT目标作为具有多说话人意识的教师信号，以明确地规范混合语音表示。为了进一步支持可变数量的说话人，我们引入了一个说话人数预测头（Talker-Count Head），该头部可以预测说话人的数量并动态选择适当的解码分支。在LibriMix数据集上的实验表明，所提出的编码器专用模型在双说话人条件下实现了与基于LLM系统的相当性能，在三说话人条件下的表现显著提升，并且实时因子（RTF）较小。
URL
https://arxiv.org/abs/2603.10587
