Abstract
Conversational automatic speech recognition remains challenging due to overlapping speech, far-field noise, and varying speaker counts. While recent LLM-based systems perform well on single-speaker benchmarks, their robustness in multi-speaker settings is unclear. We systematically compare LLM-based and modular pipeline approaches along four axes: overlap robustness, semantic fidelity, speaker count, and single- versus multi-channel input. To capture meaning-altering errors that conventional metrics miss, we introduce tcpSemER, which extends tcpWER by replacing Levenshtein distance with embedding-based semantic similarity. We further decompose tcpWER into overlapping and non-overlapping components for finer-grained analysis. Experiments across three datasets show that LLM-based systems are competitive in two-speaker settings but degrade as speaker count and overlap increase, whereas modular pipelines remain more robust.
Abstract (translated)
对话式自动语音识别仍面临重叠语音、远场噪声和说话人数变化等挑战。尽管近期基于大语言模型的系统在单说话人基准测试中表现良好，但其在多说话人场景下的鲁棒性尚不明确。我们从四个维度系统比较了基于大语言模型与模块化流水线方法：重叠鲁棒性、语义保真度、说话人数以及单通道与多通道输入。为捕捉传统指标遗漏的语义改变错误，我们提出了tcpSemER，该指标通过用基于嵌入的语义相似度替代编辑距离来扩展tcpWER。我们还进一步将tcpWER分解为重叠与非重叠部分，以实现更细粒度的分析。在三个数据集上的实验表明，基于大语言模型的系统在双说话人场景中具有竞争力，但随着说话人数增加和重叠加剧，其性能下降，而模块化流水线方法则保持更高鲁棒性。
URL
https://arxiv.org/abs/2603.22709
