Abstract
Despite recent advances, efficient and robust turn-taking detection remains a significant challenge in industrial-grade Voice AI agent deployments. Many existing systems rely solely on acoustic or semantic cues, leading to suboptimal accuracy and stability, while recent attempts to endow large language models with full-duplex capabilities require costly full-duplex data and incur substantial training and deployment overheads, limiting real-time performance. In this paper, we propose JAL-Turn, a lightweight and efficient speech-only turn-taking framework that adopts a joint acoustic-linguistic modeling paradigm, in which a cross-attention module adaptively integrates pre-trained acoustic representations with linguistic features to support low-latency prediction of hold vs shift states. By sharing a frozen ASR encoder, JAL-Turn enables turn-taking prediction to run fully in parallel with speech recognition, introducing no additional end-to-end latency or computational overhead. In addition, we introduce a scalable data construction pipeline that automatically derives reliable turn-taking labels from large-scale real-world dialogue corpora. Extensive experiments on public multilingual benchmarks and an in-house Japanese customer-service dataset show that JAL-Turn consistently outperforms strong state-of-the-art baselines in detection accuracy while maintaining superior real-time performance.
Abstract (translated)
尽管近期有所进展，但在工业级语音AI代理部署中，高效且鲁棒的轮流对话检测仍是一项重大挑战。许多现有系统仅依赖声学或语义线索，导致准确度与稳定性不足；而近期尝试赋予大语言模型全双工能力的方法，则需要昂贵的全双工数据，并产生巨大的训练与部署开销，从而限制了实时性能。本文提出JAL-Turn，一个轻量高效的纯语音轮流对话框架，采用声学-语言联合建模范式，通过跨注意力模块自适应地融合预训练声学表征与语言特征，以支持低延迟的“保持”与“转换”状态预测。通过共享一个冻结的自动语音识别编码器，JAL-Turn使轮流对话预测能与语音识别完全并行运行，不引入额外的端到端延迟或计算开销。此外，我们引入了一个可扩展的数据构建流程，能够从大规模真实对话语料中自动推导可靠的轮流对话标签。在公开多语言基准数据集和内部日语客服数据集上的大量实验表明，JAL-Turn在检测准确度上持续优于强大的最新基线方法，同时保持卓越的实时性能。
URL
https://arxiv.org/abs/2603.26515
