Abstract
Transcribing and understanding multi-speaker conversations requires speech recognition, speaker attribution, and timestamp localization. While speech LLMs excel at single-speaker tasks, multi-speaker scenarios remain challenging due to overlapping speech, backchannels, rapid turn-taking, and context window constraints. We propose Speaker-Reasoner, an end-to-end Speech LLM with agentic multi-turn temporal reasoning. Instead of single-pass inference, the model iteratively analyzes global audio structure, autonomously predicts temporal boundaries, and performs fine-grained segment analysis, jointly modeling speaker identity, gender, timestamps, and transcription. A speaker-aware cache further extends processing to audio exceeding the training context window. Trained with a three-stage progressive strategy, Speaker-Reasoner achieves consistent improvements over strong baselines on AliMeeting and AISHELL-4 datasets, particularly in handling overlapping speech and complex turn-taking.
Abstract (translated)
转写和理解多说话人对话需要语音识别、说话人归属和时间戳定位。尽管语音大语言模型在单说话人任务中表现出色，但由于重叠语音、副语言、快速轮流说话以及上下文窗口限制，多说话人场景仍具有挑战性。我们提出了Speaker-Reasoner，一个具有自主多轮时序推理能力的端到端语音大语言模型。该模型采用迭代分析而非单次推理：先全局分析音频结构，自主预测时序边界，再进行细粒度片段分析，从而联合建模说话人身份、性别、时间戳和转写内容。此外，说话人感知缓存机制可将处理扩展至超出训练上下文窗口的音频。通过三阶段渐进策略训练后，Speaker-Reasoner在AliMeeting和AISHELL-4数据集上相较于强基线取得持续提升，尤其在处理重叠语音和复杂轮流说话场景中表现突出。
URL
https://arxiv.org/abs/2604.03074
