Abstract
Automatic speech recognition (ASR) is a critical interface for human-AI interaction in gastrointestinal endoscopy, yet its reliability in real-world clinical settings is limited by domain-specific terminology and complex acoustic conditions. Here, we present EndoASR, a domain-adapted ASR system designed for real-time deployment in endoscopic workflows. We develop a two-stage adaptation strategy based on synthetic endoscopy reports, targeting domain-specific language modeling and noise robustness. In retrospective evaluation across six endoscopists, EndoASR substantially improves both transcription accuracy and clinical usability, reducing character error rate (CER) from 20.52% to 14.14% and increasing medical term accuracy (Med ACC) from 54.30% to 87.59%. In a prospective multi-center study spanning five independent endoscopy centers, EndoASR demonstrates consistent generalization under heterogeneous real-world conditions. Compared with the baseline Paraformer model, CER is reduced from 16.20% to 14.97%, while Med ACC is improved from 61.63% to 84.16%, confirming its robustness in practical deployment scenarios. Notably, EndoASR achieves a real-time factor (RTF) of 0.005, significantly faster than Whisper-large-v3 (RTF 0.055), while maintaining a compact model size of 220M parameters, enabling efficient edge deployment. Furthermore, integration with large language models demonstrates that improved ASR quality directly enhances downstream structured information extraction and clinician-AI interaction. These results demonstrate that domain-adapted ASR can serve as a reliable interface for human-AI teaming in gastrointestinal endoscopy, with consistent performance validated across multi-center real-world clinical settings.
Abstract (translated)
自动语音识别（ASR）是胃肠道内窥镜检查中人机交互的关键接口，但其在真实临床环境中的可靠性受限于领域专业术语和复杂的声学条件。本研究提出EndoASR——一个针对内窥镜工作流实时部署的领域自适应ASR系统。我们基于合成内窥镜报告开发了两阶段自适应策略，分别针对领域语言建模与噪声鲁棒性。在涵盖六位内窥镜医师的回顾性评估中，EndoASR显著提升了转写准确率与临床可用性：字错误率（CER）从20.52%降至14.14%，医学术语准确率（Med ACC）从54.30%提升至87.59%。在横跨五个独立内窥镜中心的前瞻性多中心研究中，EndoASR在 heterogeneous 真实世界条件下展现出稳定的泛化能力。相较于基线Paraformer模型，CER从16.20%降至14.97%，Med ACC从61.63%提升至84.16%，验证了其在实践部署场景中的鲁棒性。值得注意的是，EndoASR实现0.005的实时因子（RTF），显著快于Whisper-large-v3（RTF 0.055），同时保持2.2亿参数的紧凑模型规模，支持高效边缘部署。此外，与大语言模型的集成表明，ASR质量的提升直接增强了下游结构化信息提取与医工交互效能。这些结果证明，领域自适应ASR可成为胃肠道内窥镜人机协作的可靠接口，其跨多中心真实临床场景的一致性能已获验证。
URL
https://arxiv.org/abs/2604.01705
