Speech_Recognition
-
LoASR-Bench: Evaluating Large Speech Language Models on Low-Resource Automatic Speech Recognition Across Language Families
2026-03-20 15:26:34Jianan Chen, Xiaoxue Gao, Tatsuya Kawahara, Nancy F. ChenAbstract
Large language models (LLMs) have driven substantial advances in speech language models (SpeechLMs), yielding strong performance in automatic speech recognition (ASR) under high-resource conditions. However, existing benchmarks predominantly focus on high-resource languages, leaving the ASR behavior of SpeechLMs in low-resource languages insufficiently understood. This gap is critical, as practical ASR systems must reliably support low-resource languages and generalize across diverse language families, and it directly hinders the deployment of SpeechLM-based ASR in real-world multilingual scenarios. As a result, it is essential to evaluate SpeechLMs on low-resource languages to ensure their generalizability across different language families. To address this problem, we propose \textbf{LoASR-Bench}, a comprehensive benchmark designed to evaluate \textbf{lo}w-resource \textbf{a}utomatic \textbf{s}peech \textbf{r}ecognition (\textbf{ASR}) of the latest SpeechLMs across diverse language families. LoASR-Bench comprises 25 languages from 9 language families, featuring both Latin and non-Latin scripts, enabling cross-linguistic and cross-script assessment of ASR performance of current SpeechLMs. Experimental results highlight the limitations of the latest SpeechLMs in handling real-world low-resource languages.
Abstract (translated)
大语言模型（LLMs）推动了语音语言模型（SpeechLMs）的显著进步，在高资源条件下自动语音识别（ASR）任务中展现出强劲性能。然而，现有基准测试主要关注高资源语言，导致SpeechLMs在低资源语言中的ASR行为尚未得到充分理解。这一空白至关重要，因为实际ASR系统必须可靠地支持低资源语言，并跨不同语系实现泛化，这直接阻碍了基于SpeechLM的ASR在真实多语言场景中的部署。因此，为评估SpeechLMs在不同语系间的泛化能力，必须在低资源语言上进行测试。针对这一问题，我们提出了\textbf{LoASR-Bench}，这是一个用于评估最新SpeechLMs在跨语系\textbf{低资源自动语音识别（ASR）}表现的综合基准。LoASR-Bench涵盖9个语系的25种语言，包含拉丁与非拉丁文字系统，可对当前SpeechLMs的ASR性能进行跨语言与跨文字评估。实验结果表明，最新SpeechLMs在处理真实低资源语言时存在明显局限。
URL
https://arxiv.org/abs/2603.20042
PDF
-
Impact of automatic speech recognition quality on Alzheimer's disease detection from spontaneous speech: a reproducible benchmark study with lexical modeling and statistical validation
2026-03-18 19:46:29Himadri SamantaAbstract
Early detection of Alzheimer's disease from spontaneous speech has emerged as a promising non-invasive screening approach. However, the influence of automatic speech recognition (ASR) quality on downstream clinical language modeling remains insufficiently understood. In this study, we investigate Alzheimer's disease detection using lexical features derived from Whisper ASR transcripts on the ADReSSo 2021 diagnosis dataset. We evaluate interpretable machine-learning models, including Logistic Regression and Linear Support Vector Machines, using TF-IDF text representations under repeated 5x5 stratified cross-validation. Our results demonstrate that transcript quality has a statistically significant impact on classification performance. Models trained on Whisper-small transcripts consistently outperform those using Whisper-base transcripts, achieving balanced accuracy above 0.7850 with Linear SVM. Paired statistical testing confirms that the observed improvements are significant. Importantly, classifier complexity contributes less to performance variation than ASR transcription quality. Feature analysis reveals that cognitively normal speakers produce more semantically precise object- and scene-descriptive language, whereas Alzheimer's speech is characterized by vagueness, discourse markers, and increased hesitation patterns. These findings suggest that high-quality ASR can enable simple, interpretable lexical models to achieve competitive Alzheimer's detection performance without explicit acoustic modeling. The study provides a reproducible benchmark pipeline and highlights ASR selection as a critical modeling decision in clinical speech-based artificial intelligence systems.
Abstract (translated)
从自发语音中早期检测阿尔茨海默病已成为一种有前景的非侵入性筛查方法。然而，自动语音识别（ASR）质量对下游临床语言建模的影响仍缺乏充分理解。本研究基于ADReSSo 2021诊断数据集，利用Whisper ASR转写文本衍生的词汇特征进行阿尔茨海默病检测研究。我们采用TF-IDF文本表示，在重复5×5分层交叉验证框架下评估可解释机器学习模型（包括逻辑回归与线性支持向量机）。结果表明，转写质量对分类性能具有统计显著影响：基于Whisper-small转写文本训练的模型持续优于Whisper-base版本，其中线性支持向量机达到0.7850以上的平衡准确率。配对统计检验证实观察到的改进具有显著性。重要的是，分类器复杂度对性能变异的影响小于ASR转写质量。特征分析显示，认知正常者产出更多语义精确的物体与场景描述性语言，而阿尔茨海默病语音则表现为模糊表达、话语标记及犹豫模式增加。这些发现表明，高质量ASR可使简单的可解释词汇模型无需显式声学建模即可实现具有竞争力的阿尔茨海默病检测性能。本研究提供了可复现的基准流程，并强调ASR选择是临床语音人工智能系统的关键建模决策。
URL
https://arxiv.org/abs/2603.18239
PDF
-
Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognition
2026-03-18 10:04:50Yuxiang Mei, Delai Qiu, Shengping Liu, Jiaen Liang, Yanhua LongAbstract
Speech Large Language Models (Speech-LLMs) have emerged as a powerful approach for automatic speech recognition (ASR) by aligning speech encoders with large language models. However, adapting these systems to multilingual settings with imbalanced data distributions remains challenging. In such scenarios, a stability-plasticity dilemma often arises: fully shared Parameter-Efficient Fine-Tuning (PEFT) can cause negative inter-lingual interference for under-represented languages, while fully language-specific tuning limits the cross-lingual beneficial knowledge transfer needed for low-resource tasks. To address this, we propose Zipper-LoRA, a novel rank-level decoupling framework with three variants (Static, Hard, and Soft) that dynamically synthesizes LoRA updates from shared and language-specific subspaces. By using a lightweight language-conditioned router, Zipper-LoRA dynamically controls the contribution of each subspace at the LoRA rank level, enabling fine-grained sharing where languages are compatible and strict decoupling when conflicts occur. To further stabilize optimization under imbalanced data, we propose a two-stage training strategy with an Initial-B warm start that significantly accelerates convergence. Experiments on a 12-language mixed-resource setting show that Zipper-LoRA consistently outperforms both fully shared and independent baselines, particularly in extremely low-resource scenarios. Moreover, we demonstrate that these gains are robust across both chunked and non-chunked encoder configurations, confirming the framework's reliability for practical, large-scale multilingual ASR. Our code and data will be available at this https URL for reproducibility.
Abstract (translated)
URL
https://arxiv.org/abs/2603.17558
PDF
-
A Proactive EMR Assistant for Doctor-Patient Dialogue: Streaming ASR, Belief Stabilization, and Preliminary Controlled Evaluation
2026-03-18 05:54:01Zhenhai Pan, Yan Liu, Jia YouAbstract
Most dialogue-based electronic medical record (EMR) systems still behave as passive pipelines: transcribe speech, extract information, and generate the final note after the consultation. That design improves documentation efficiency, but it is insufficient for proactive consultation support because it does not explicitly address streaming speech noise, missing punctuation, unstable diagnostic belief, objectification quality, or measurable next-action gains. We present an end-to-end proactive EMR assistant built around streaming speech recognition, punctuation restoration, stateful extraction, belief stabilization, objectified retrieval, action planning, and replayable report generation. The system is evaluated in a preliminary controlled setting using ten streamed doctor-patient dialogues and a 300-query retrieval benchmark aggregated across dialogues. The full system reaches state-event F1 of 0.84, retrieval Recall@5 of 0.87, and end-to-end pilot scores of 83.3% coverage, 81.4% structural completeness, and 80.0% risk recall. Ablations further suggest that punctuation restoration and belief stabilization may improve downstream extraction, retrieval, and action selection within this pilot. These results were obtained under a controlled simulated pilot setting rather than broad deployment claims, and they should not be read as evidence of clinical deployment readiness, clinical safety, or real-world clinical utility. Instead, they suggest that the proposed online architecture may be technically coherent and directionally supportive under tightly controlled pilot conditions. The present study should be read as a pilot concept demonstration under tightly controlled pilot conditions rather than as evidence of clinical deployment readiness or clinical generalizability.
Abstract (translated)
URL
https://arxiv.org/abs/2604.13059
PDF
-
RECOVER: Robust Entity Correction via agentic Orchestration of hypothesis Variants for Evidence-based Recovery
2026-03-17 11:48:12Abhishek Kumar, Aashraya SachdevaAbstract
Entity recognition in Automatic Speech Recognition (ASR) is challenging for rare and domain-specific terms. In domains such as finance, medicine, and air traffic control, these errors are costly. If the entities are entirely absent from the ASR output, post-ASR correction becomes difficult. To address this, we introduce RECOVER, an agentic correction framework that serves as a tool-using agent. It leverages multiple hypotheses as evidence from ASR, retrieves relevant entities, and applies Large Language Model (LLM) correction under constraints. The hypotheses are used using different strategies, namely, 1-Best, Entity-Aware Select, Recognizer Output Voting Error Reduction (ROVER) Ensemble, and LLM-Select. Evaluated across five diverse datasets, it achieves 8-46% relative reductions in entity-phrase word error rate (E-WER) and increases recall by up to 22 percentage points. The LLM-Select achieves the best overall performance in entity correction while maintaining overall WER.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16411
PDF
-
Over-the-air White-box Attack on the Wav2Vec Speech Recognition Neural Network
2026-03-17 11:42:10Protopopov AlexeyAbstract
Automatic speech recognition systems based on neural networks are vulnerable to adversarial attacks that alter transcriptions in a malicious way. Recent works in this field have focused on making attacks work in over-the-air scenarios, however such attacks are typically detectable by human hearing, limiting their potential applications. In the present work we explore different approaches of making over-the-air attacks less detectable, as well as the impact these approaches have on the attacks' effectiveness.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16972
PDF
-
Is Semi-Automatic Transcription Useful in Corpus Creation? Preliminary Considerations on the KIParla Corpus
2026-03-17 08:44:37Martina Simonotti, Ludovica Pannitto, Eleonora Zucchini, Silvia Ballar\`e, Caterina MauriAbstract
This paper analyses the implementation of Automatic Speech Recognition (ASR) into the transcription workflow of the KIParla corpus, a resource of spoken Italian. Through a two-phase experiment, 11 expert and novice transcribers produced both manual and ASR-assisted transcriptions of identical audio segments across three different types of conversation, which were subsequently analyzed through a combination of statistical modeling, word-level alignment and a series of annotation-based metrics. Results show that ASR-assisted workflows can increase transcription speed but do not consistently improve overall accuracy, with effects depending on multiple factors such as workflow configuration, conversation type and annotator experience. Analyses combining alignment-based metrics, descriptive statistics and statistical modeling provide a systematic framework to monitor transcription behavior across annotators and workflows. Despite limitations, ASR-assisted transcription, potentially supported by task-specific fine-tuning, could be integrated into the KIParla transcription workflow to accelerate corpus creation without compromising transcription quality.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16258
PDF
-
Polyglot-Lion: Efficient Multilingual ASR for Singapore via Balanced Fine-Tuning of Qwen3-ASR
2026-03-17 07:09:42Quy-Anh Dang, Chris NgoAbstract
We present Polyglot-Lion, a family of compact multilingual automatic speech recognition (ASR) models tailored for the linguistic landscape of Singapore, covering English, Mandarin, Tamil, and Malay. Our models are obtained by fine-tuning Qwen3-ASR-0.6B and Qwen3-ASR-1.7B exclusively on publicly available speech corpora, using a balanced sampling strategy that equalizes the number of training utterances per language and deliberately omits language-tag conditioning so that the model learns to identify languages implicitly from audio. On 12 benchmarks spanning the four target languages, Polyglot-Lion-1.7B achieves an average error rate of 14.85, competitive with MERaLiON-2-10B-ASR (14.32) - a model 6x larger - while incurring a training cost of \$81 on a single RTX PRO 6000 GPU compared to \$18,862 for the 128-GPU baseline. Inference throughput is approximately 20x faster than MERaLiON at 0.10 s/sample versus 2.02 s/sample. These results demonstrate that linguistically balanced fine-tuning of moderate-scale pretrained models can yield deployment-ready multilingual ASR at a fraction of the cost of larger specialist systems.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16184
PDF
-
Tagarela - A Portuguese speech dataset from podcasts
2026-03-16 14:18:55Frederico Santos de Oliveira, Lucas Rafael Stefanel Gris, Alef Iury Siqueira Ferreira, Augusto Seben da Rosa, Alexandre Costa Ferro Filho, Edresson Casanova, Christopher Dane Shulby, Rafael Teixeira Sousa, Diogo Fernandes Costa Silva, Anderson da Silva Soares, Arlindo Rodrigues Galv\~ao FilhoAbstract
Despite significant advances in speech processing, Portuguese remains under-resourced due to the scarcity of public, large-scale, and high-quality datasets. To address this gap, we present a new dataset, named TAGARELA, composed of over 8,972 hours of podcast audio, specifically curated for training automatic speech recognition (ASR) and text-to-speech (TTS) models. Notably, its scale rivals English's GigaSpeech (10kh), enabling state-of-the-art Portuguese models. To ensure data quality, the corpus was subjected to an audio pre-processing pipeline and subsequently transcribed using a mixed strategy: we applied ASR models that were previously trained on high-fidelity transcriptions generated by proprietary APIs, ensuring a high level of initial accuracy. Finally, to validate the effectiveness of this new resource, we present ASR and TTS models trained exclusively on our dataset and evaluate their performance, demonstrating its potential to drive the development of more robust and natural speech technologies for Portuguese. The dataset is released publicly, available at this https URL, to foster the development of robust speech technologies.
Abstract (translated)
URL
https://arxiv.org/abs/2603.15326
PDF
-
Two-Stage Adaptation for Non-Normative Speech Recognition: Revisiting Speaker-Independent Initialization for Personalization
2026-03-16 13:27:53Shan Jiang, Jiawen Qi, Chuanbing Huo, Yingqiang Gao, Qinyu ChenAbstract
Personalizing automatic speech recognition (ASR) systems for non-normative speech, such as dysarthric and aphasic speech, is challenging. While speaker-specific fine-tuning (SS-FT) is widely used, it is typically initialized directly from a generic pre-trained model. Whether speaker-independent adaptation provides a stronger initialization prior under such mismatch remains unclear. In this work, we propose a two-stage adaptation framework consisting of speaker-independent fine-tuning (SI-FT) on multi-speaker non-normative data followed by SS-FT, and evaluate it through a controlled comparison with direct SS-FT under identical per-speaker conditions. Experiments on AphasiaBank and UA-Speech with Whisper-Large-v3 and Qwen3-ASR, alongside evaluation on typical-speech datasets TED-LIUM v3 and FLEURS, show that two-stage adaptation consistently improves personalization while maintaining manageable out-of-domain (OOD) trade-offs.
Abstract (translated)
URL
https://arxiv.org/abs/2603.15261
PDF
-
Vietnamese Automatic Speech Recognition: A Revisit
2026-03-16 03:23:54Thi Vu, Linh The Nguyen, Dat Quoc NguyenAbstract
Automatic Speech Recognition (ASR) performance is heavily dependent on the availability of large-scale, high-quality datasets. For low-resource languages, existing open-source ASR datasets often suffer from insufficient quality and inconsistent annotation, hindering the development of robust models. To address these challenges, we propose a novel and generalizable data aggregation and preprocessing pipeline designed to construct high-quality ASR datasets from diverse, potentially noisy, open-source sources. Our pipeline incorporates rigorous processing steps to ensure data diversity, balance, and the inclusion of crucial features like word-level timestamps. We demonstrate the effectiveness of our methodology by applying it to Vietnamese, resulting in a unified, high-quality 500-hour dataset that provides a foundation for training and evaluating state-of-the-art Vietnamese ASR systems. Our project page is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.14779
PDF
-
Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition
2026-03-12 15:22:27Umberto Cappellazzo, Stavros Petridis, Maja PanticAbstract
Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework using Shapley values to analyze modality contributions in AVSR. Through experiments on six models across two benchmarks and varying SNR levels, we introduce three analyses: Global SHAP for overall modality balance, Generative SHAP for contribution dynamics during decoding, and Temporal Alignment SHAP for input-output correspondence. Our findings reveal that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation. Modality balance evolves during generation, temporal alignment holds under noise, and SNR is the dominant factor driving modality weighting. These findings expose a persistent audio bias, motivating ad-hoc modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.
Abstract (translated)
音频-视觉语音识别（AVSR）利用声学和视觉信息，在噪声环境下进行稳健的语音识别。然而，模型如何平衡这些模态仍不清楚。我们提出了Dr. SHAP-AV框架，该框架使用Shapley值来分析AVSR中模式贡献情况。通过在两个基准测试集上对六种不同模型以及各种信噪比（SNR）水平进行实验，我们引入了三种分析方法：全局SHAP用于整体模态平衡、生成式SHAP用于解码过程中的贡献动态变化、时间对齐SHAP用于输入输出对应关系。我们的研究发现表明，在噪声环境下，模型倾向于依赖视觉信息，但在严重退化情况下仍保持较高的声学信息贡献水平。模式间的平衡在生成过程中发生变化，而时间上的对齐性在噪声下依然存在，信噪比是驱动模式权重的主要因素。 这些发现揭示了持续存在的音频偏见，这促使我们需要采用临时的模式权重机制，并将基于Shapley值的归因方法作为AVSR诊断的标准。
URL
https://arxiv.org/abs/2603.12046
PDF
-
Continued Pretraining for Low-Resource Swahili ASR: Achieving State-of-the-Art Performance with Minimal Labeled Data
2026-03-11 23:45:06Hillary Mutisya, John MuganeAbstract
We investigate continued pretraining (CPT) for adapting wav2vec2-bert-2.0 to Swahili automatic speech recognition (ASR). Our approach combines unlabeled audio with limited labeled data through pseudo-labeled CPT followed by supervised finetuning. With 20,000 labeled samples, we achieve 3.24% WER on Common Voice Swahili-an 82% relative improvement over the baseline. This result surpasses the best previously reported academic system (8.3% WER from XLS-R) by 61% relative improvement. We provide concrete data requirements and a replicable methodology applicable to other low-resource languages.
Abstract (translated)
我们研究了用于将wav2vec2-bert-2.0模型适应斯瓦希里语自动语音识别（ASR）的持续预训练（CPT）。我们的方法结合了未标记音频和有限标签数据，通过伪标注CPT然后进行监督微调。使用20,000个标注样本，在Common Voice 斯瓦希里语测试集上实现了3.24%的WER，相较于基线模型相对改进了82%。这一结果超越了之前最佳的学术系统（XLS-R系统的8.3% WER），相对提高了61%。我们提供了具体的数据需求和可重复的方法论，适用于其他低资源语言。
URL
https://arxiv.org/abs/2603.11378
PDF
-
Learnable Pulse Accumulation for On-Device Speech Recognition: How Much Attention Do You Need?
2026-03-11 20:49:32Yakov Pyotr ShkolnikovAbstract
Self-attention scales quadratically with sequence length, limiting transformer-based speech models on edge devices. We introduce the Learnable Pulse Accumulator (LPA), an O(n) replacement that substitutes key-query dot products with learned gating functions: content-dependent rectangular pulses, periodic windows, and position-dependent basis functions. An MSE diagnostic sweep determines per-layer replacement difficulty and ordering. Replacing 8 of 12 wav2vec2-base layers yields 10.61% word error rate (WER) on LibriSpeech test-clean, +7.24 percentage points (pp) over the 3.37% baseline, with 3.27x speedup at 120s audio on Apple M4 Pro via an optimized MLX inference path. Cross-domain validation on SepFormer speech enhancement shows all 16 intra-chunk attention layers can be replaced without collapse, suggesting the depth wall arises from linguistic computation rather than an LPA limitation. LPA's near-binary gates at inference enable dense GPU computation with no CPU-GPU synchronization, and all operations map to mobile neural accelerators.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16922
PDF
-
Huntington Disease Automatic Speech Recognition with Biomarker Supervision
2026-03-11 18:00:02Charles L. Wang, Cady Chen, Ziwei Gong, Julia HirschbergAbstract
Automatic speech recognition (ASR) for pathological speech remains underexplored, especially for Huntington's disease (HD), where irregular timing, unstable phonation, and articulatory distortion challenge current models. We present a systematic HD-ASR study using a high-fidelity clinical speech corpus not previously used for end-to-end ASR training. We compare multiple ASR families under a unified evaluation, analyzing WER as well as substitution, deletion, and insertion patterns. HD speech induces architecture-specific error regimes, with Parakeet-TDT outperforming encoder-decoder and CTC baselines. HD-specific adaptation reduces WER from 6.99% to 4.95% and we also propose a method for using biomarker-based auxiliary supervision and analyze how error behavior is reshaped in severity-dependent ways rather than uniformly improving WER. We open-source all code and models.
Abstract (translated)
自动语音识别（ASR）在处理病理性的言语，尤其是亨廷顿病（HD）患者的言语时，仍然是一项未被充分研究的领域。亨廷顿病患者常表现出不规则的时间感、不稳定的声音产生和发音扭曲等问题，这对现有的模型构成了挑战。 我们提出了一项系统性针对亨廷顿病ASR的研究，使用了一个此前从未用于端到端ASR训练的高保真临床语音语料库。我们在统一评估框架下对比了多个ASR家族，并分析了WER（单词错误率）、替换、删除和插入模式等指标。 研究发现，亨廷顿病患者的言语会导致特定架构下的错误模式，其中Parakeet-TDT模型在性能上超过了编码器-解码器及CTC基准模型。针对亨廷顿病的特异性适应性调整将WER从6.99%降低到4.95%，我们还提出了一种基于生物标志物辅助监督的方法，并分析了如何以依赖疾病严重程度的方式重塑错误行为，而不是单纯地提高WER。 我们的研究代码和模型现已开源。
URL
https://arxiv.org/abs/2603.11168
PDF
-
Uni-ASR: Unified LLM-Based Architecture for Non-Streaming and Streaming Automatic Speech Recognition
2026-03-11 14:24:45Yinfeng Xia, Jian Tang, Junfeng Hou, Gaopeng Xu, Haitao YaoAbstract
Although the deep integration of the Automatic Speech Recognition (ASR) system with Large Language Models (LLMs) has significantly improved accuracy, the deployment of such systems in low-latency streaming scenarios remains challenging. In this paper, we propose Uni-ASR, a unified framework based on LLMs that integrates both non-streaming and streaming speech recognition capabilities. We propose a joint training paradigm that enables the system to seamlessly transition between two recognition modes without any architectural modifications. Furthermore, we introduce a context-aware training paradigm and a co-designed fallback decoding strategy, which can enhance streaming recognition accuracy without introducing additional latency. The experimental results demonstrate that Uni-ASR not only achieves competitive performance within non-streaming mode, but also demonstrates strong effectiveness in streaming scenarios under diverse latency constraints.
Abstract (translated)
尽管自动语音识别（ASR）系统与大型语言模型（LLM）的深度集成显著提高了准确性，但在低延迟流式场景中部署此类系统仍然具有挑战性。在本文中，我们提出了Uni-ASR，这是一种基于LLM的统一框架，集成了非流式和流式语音识别能力。我们提出了一种联合训练范式，使系统能够在两种识别模式之间无缝切换，而无需进行架构修改。此外，我们引入了上下文感知训练范式以及协同设计的退避解码策略，这可以在不增加额外延迟的情况下提高流式识别准确性。实验结果表明，Uni-ASR不仅在非流式模式下实现了竞争性性能，还在不同延迟约束条件下的流式场景中表现出强大的有效性。
URL
https://arxiv.org/abs/2603.11123
PDF
-
AlphaFlowTSE: One-Step Generative Target Speaker Extraction via Conditional AlphaFlow
2026-03-11 12:14:25Duojia Li, Shuhan Zhang, Zihan Qian, Wenxuan Wu, Shuai Wang, Qingyang Hong, Lin Li, Haizhou LiAbstract
In target speaker extraction (TSE), we aim to recover target speech from a multi-talker mixture using a short enrollment utterance as reference. Recent studies on diffusion and flow-matching generators have improved target-speech fidelity. However, multi-step sampling increases latency, and one-step solutions often rely on a mixture-dependent time coordinate that can be unreliable for real-world conversations. We present AlphaFlowTSE, a one-step conditional generative model trained with a Jacobian-vector product (JVP)-free AlphaFlow objective. AlphaFlowTSE learns mean-velocity transport along a mixture-to-target trajectory starting from the observed mixture, eliminating auxiliary mixing-ratio prediction, and stabilizes training by combining flow matching with an interval-consistency teacher-student target. Experiments on Libri2Mix and REAL-T confirm that AlphaFlowTSE improves target-speaker similarity and real-mixture generalization for downstream automatic speech recognition (ASR).
Abstract (translated)
在目标说话人提取（TSE）中，我们的目标是从多说话人的混合信号中恢复出目标语音，并使用短的注册语句作为参考。最近关于扩散和流匹配生成器的研究提高了目标语音的质量。然而，多步骤采样增加了延迟，而一步解决方案通常依赖于一个对现实对话可能不可靠的混合时间坐标。我们提出了AlphaFlowTSE，这是一个通过无雅可比向量积（JVP）自由的AlphaFlow目标进行训练的一步式条件生成模型。AlphaFlowTSE从观察到的混合信号开始学习均值速度传输沿混合至目标轨迹，消除了辅助混音比率预测，并结合流匹配与区间一致性师生目标来稳定训练。在Libri2Mix和REAL-T上的实验表明，AlphaFlowTSE提高了目标说话人的相似度以及对下游自动语音识别（ASR）的真实混合泛化能力。
URL
https://arxiv.org/abs/2603.10701
PDF
-
Distilling LLM Semantic Priors into Encoder-Only Multi-Talker ASR with Talker-Count Routing
2026-03-11 09:43:49Hao Shi, Yusuke Fujita, Roman Koshkin, Mengjie Zhao, Yuan Gao, Lianbo Liu, Yui SudoAbstract
Large language models (LLMs) provide strong semantic priors that can improve multi-talker automatic speech recognition (MT-ASR), but using an LLM as an autoregressive decoder is computationally expensive and remains fragile under heavy overlap. In this paper, we propose an encoder-only MT-ASR framework that adapts an LLM to multi-talker conditioning and distills its semantic guidance into the encoder during training, while retaining fast CTC-style decoding at inference. Our model employs a post-encoder separator with serialized CTC to produce talker-ordered transcripts, and leverages an adapted LLM-based SOT objective as a multi-talker-aware teacher signal to explicitly regularize mixed-speech representations. To further support variable numbers of talkers, we introduce a Talker-Count Head that predicts the talker count and dynamically selects the appropriate decoding branch. Experiments on LibriMix show that the proposed encoder-only model achieves comparable performance to LLM-based systems in the two-talker condition, while delivering significant improvements in the three-talker condition with significant small RTF.
Abstract (translated)
大型语言模型（LLM）提供了强大的语义先验，可以改善多说话人自动语音识别（MT-ASR），但将LLM用作自回归解码器在计算上是昂贵的，并且在重叠严重时仍然不稳定。本文提出了一种编码器专用的多说话人语音识别框架，该框架使大型语言模型适应于多说话人的条件，并在训练过程中将其语义指导蒸馏到编码器中，同时保持快速的CTC风格解码进行推理。我们的模型采用了后编码分割器和序列化的CTC来生成有序说话人记录，并利用了基于改编LLM的SOT目标作为具有多说话人意识的教师信号，以明确地规范混合语音表示。为了进一步支持可变数量的说话人，我们引入了一个说话人数预测头（Talker-Count Head），该头部可以预测说话人的数量并动态选择适当的解码分支。在LibriMix数据集上的实验表明，所提出的编码器专用模型在双说话人条件下实现了与基于LLM系统的相当性能，在三说话人条件下的表现显著提升，并且实时因子（RTF）较小。
URL
https://arxiv.org/abs/2603.10587
PDF
-
Synthetic Data Domain Adaptation for ASR via LLM-based Text and Phonetic Respelling Augmentation
2026-03-11 06:56:56Natsuo Yamashita, Koichi Nagatsuka, Hiroaki Kokubo, Kota Dohi, Tuan Vu HoAbstract
End-to-end automatic speech recognition often degrades on domain-specific data due to scarce in-domain resources. We propose a synthetic-data-based domain adaptation framework with two contributions: (1) a large language model (LLM)-based text augmentation pipeline with a filtering strategy that balances lexical diversity, perplexity, and domain-term coverage, and (2) phonetic respelling augmentation (PRA), a novel method that introduces pronunciation variability through LLM-generated orthographic pseudo-spellings. Unlike conventional acoustic-level methods such as SpecAugment, PRA provides phonetic diversity before speech synthesis, enabling synthetic speech to better approximate real-world variability. Experimental results across four domain-specific datasets demonstrate consistent reductions in word error rate, confirming that combining domain-specific lexical coverage with realistic pronunciation variation significantly improves ASR robustness.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16920
PDF
-
FireRedASR2S: A State-of-the-Art Industrial-Grade All-in-One Automatic Speech Recognition System
2026-03-11 05:11:06Kaituo Xu, Yan Jia, Kai Huang, Junjie Chen, Wenpeng Li, Kun Liu, Feng-Long Xie, Xu Tang, Yao HuAbstract
We present FireRedASR2S, a state-of-the-art industrial-grade all-in-one automatic speech recognition (ASR) system. It integrates four modules in a unified pipeline: ASR, Voice Activity Detection (VAD), Spoken Language Identification (LID), and Punctuation Prediction (Punc). All modules achieve SOTA performance on the evaluated benchmarks: FireRedASR2: An ASR module with two variants, FireRedASR2-LLM (8B+ parameters) and FireRedASR2-AED (1B+ parameters), supporting speech and singing transcription for Mandarin, Chinese dialects and accents, English, and code-switching. Compared to FireRedASR, FireRedASR2 delivers improved recognition accuracy and broader dialect and accent coverage. FireRedASR2-LLM achieves 2.89% average CER on 4 public Mandarin benchmarks and 11.55% on 19 public Chinese dialects and accents benchmarks, outperforming competitive baselines including Doubao-ASR, Qwen3-ASR, and Fun-ASR. FireRedVAD: An ultra-lightweight module (0.6M parameters) based on the Deep Feedforward Sequential Memory Network (DFSMN), supporting streaming VAD, non-streaming VAD, and multi-label VAD (mVAD). On the FLEURS-VAD-102 benchmark, it achieves 97.57% frame-level F1 and 99.60% AUC-ROC, outperforming Silero-VAD, TEN-VAD, FunASR-VAD, and WebRTC-VAD. FireRedLID: An Encoder-Decoder LID module supporting 100+ languages and 20+ Chinese dialects and accents. On FLEURS (82 languages), it achieves 97.18% utterance-level accuracy, outperforming Whisper and SpeechBrain. FireRedPunc: A BERT-style punctuation prediction module for Chinese and English. On multi-domain benchmarks, it achieves 78.90% average F1, outperforming FunASR-Punc (62.77%). To advance research in speech processing, we release model weights and code at this https URL.
Abstract (translated)
我们介绍FireRedASR2S，这是一个最先进的工业级一体化自动语音识别（ASR）系统。它在一个统一的流水线中集成了四个模块：ASR、语音活动检测（VAD）、口语语言识别（LID）和标点预测（Punc）。所有模块在评估基准上均达到了最先进水平（SOTA）： - FireRedASR2: 包含两个变体，FireRedASR2-LLM（8B+参数）和支持语音和演唱转写的FireRedASR2-AED（1B+参数），支持普通话、中国方言及口音、英语以及代码切换。与前代相比，FireRedASR2在识别准确性和更广泛的方言及口音覆盖范围方面都有所改进。FireRedASR2-LLM在中国语料库的四个公共基准上平均CER为2.89%，在19个公共中国方言和口音基准上的平均CER为11.55%，优于包括Doubao-ASR、Qwen3-ASR和Fun-ASR在内的竞争基线。 - FireRedVAD：一个基于深度前馈序列记忆网络（DFSMN）的超轻量级模块，支持流式语音活动检测（streaming VAD）、非流式语音活动检测（non-streaming VAD）以及多标签语音活动检测（mVAD）。在FLEURS-VAD-102基准上，它实现了97.57%的帧级F1和99.60%AUC-ROC，优于Silero-VAD、TEN-VAD、FunASR-VAD及WebRTC-VAD。 - FireRedLID：一个支持超过100种语言和20多种中国方言及口音的编码器解码器模块。在FLEURS（82种语言）上，它实现了97.18%的句子级准确率，优于Whisper和SpeechBrain。 - FireRedPunc：针对中文和英文的BERT风格标点预测模块，在多领域基准上平均F1为78.90%，优于FunASR-Punc（62.77%）。 为了推进语音处理领域的研究，我们将在[此链接](https://this-url.com)发布模型权重及代码。
URL
https://arxiv.org/abs/2603.10420
PDF
