Abstract
Dysarthric speech reconstruction (DSR) typically employs a cascaded system that combines automatic speech recognition (ASR) and sentence-level text-to-speech (TTS) to convert dysarthric speech into normally-prosodied speech. However, dysarthric individuals often speak more slowly, leading to excessively long response times in such systems, rendering them impractical in long-speech scenarios. Cascaded DSR systems based on streaming ASR and incremental TTS can help reduce latency. However, patients with differing dysarthria severity exhibit substantial pronunciation variability for the same text, resulting in poor robustness of ASR and limiting the intelligibility of reconstructed speech. In addition, incremental TTS suffers from poor prosodic feature prediction due to a limited receptive field. In this study, we propose an end-to-end simultaneous DSR system with two key innovations: 1) A frame-level adaptor module is introduced to bridge ASR and TTS. By employing explicit-implicit semantic information fusion and joint module training, it enhances the error tolerance of TTS to ASR outputs. 2) A multiple wait-k autoregressive TTS module is designed to mitigate prosodic degradation via multi-view knowledge distillation. Our system has an average response time of 1.03 seconds on Tesla A100, with an average real-time factor (RTF) of 0.71. On the UASpeech dataset, it attains a mean opinion score (MOS) of 4.67 and demonstrates a 54.25% relative reduction in word error rate (WER) compared to the state-of-the-art. Our demo is available at: this https URL
Abstract (translated)
失语症语音重建（DSR）通常采用级联系统，结合自动语音识别（ASR）和句子级别的文本到语音合成（TTS），将失语症患者的语音转换为正常的音调语音。然而，失语症患者往往说话速度较慢，导致此类系统的响应时间过长，在处理长时间对话时显得不切实际。基于流式ASR和增量TTS的级联DSR系统有助于减少延迟。但是，患有不同程度失语症的病人对于同一文本发音差异显著，这会导致ASR的鲁棒性较差，并限制了重建语音的可理解性。此外，由于感受野有限，增量TTS在预测音调特征方面表现不佳。 在这项研究中，我们提出了一种端到端的同时DSR系统，具有两个关键创新：1）引入了一个帧级适配器模块来连接ASR和TTS。通过采用显式-隐式语义信息融合及联合模块训练，它增强了TTS对ASR输出的错误容忍度；2）设计了一种多等待-k自回归TTS模块，通过多视角知识蒸馏缓解音调退化问题。 我们的系统在Tesla A100上的平均响应时间为1.03秒，实时因子（RTF）平均为0.71。在UASpeech数据集上，它获得了4.67的平均意见得分（MOS），并且相较于最先进的方法，单词错误率（WER）相对降低了54.25%。 我们的演示可在以下链接查看：[此处提供链接]
URL
https://arxiv.org/abs/2603.01382
