Speech_Recognition
-
BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding
2026-03-04 16:11:49Tommaso Boccato, Michal Olak, Matteo FerranteAbstract
Decoding continuous speech from intracortical recordings is a central challenge for brain-computer interfaces (BCIs), with transformative potential for individuals with conditions that impair their ability to speak. While recent microelectrode array (MEA) decoders achieve impressive accuracy, their performance is fundamentally limited by the small size of existing datasets, they remain brittle to session-to-session variability, and their ability to generalize across participants remains unexplored. We introduce BrainWhisperer, a neural speech decoder that integrates high-resolution MEA recordings with a large pretrained automatic speech recognition (ASR) model. Building on interpretability findings showing that Whisper's encoder learns phoneme-selective representations with localized attention, we train a customized version of Whisper, modified to process neural features, using a hybrid objective that combines CTC loss on phonemes--predicted from the third encoder layer--and cross-entropy loss on word tokens. We introduce domain-informed modifications including windowed self-attention to capture articulatory continuity, hierarchical month/day-specific low-rank projections to address non-stationarity, and subject-specific embedders enabling cross-subject training. Evaluated on a publicly available MEA dataset (Card et al.), BrainWhisperer matches or outperforms prior state-of-the-art decoders. Critically, cross-dataset training improves performance even on individual datasets without fine-tuning, demonstrating unprecedented generalization. The model supports dual decoding paths: a high-accuracy phoneme-based path with external language model rescoring, and a fast direct text generation path enabling sub-100ms inference with minimal hardware requirements.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13321
PDF
-
Robust LLM-based Audio-Visual Speech Recognition with Sparse Modality Alignment and Visual Unit-Guided Refinement
2026-03-04 07:47:30Fei Su, Cancan Li, Juan Liu, Wei Ju, Hongbin Suo, Ming LiAbstract
Audio-Visual Speech Recognition (AVSR) integrates acoustic and visual information to enhance robustness in adverse acoustic conditions. Recent advances in Large Language Models (LLMs) have yielded competitive automatic speech recognition performance and shown effectiveness for AVSR. However, prior approaches project audio and visual features independently or apply shallow fusion, limiting cross-modal alignment and complementary exchange while increasing the LLM's computational load. To address this, we propose AVUR-LLM, an LLM-based Audio-Visual Speech Recognition via Sparse Modality Alignment and Visual Unit-Guided Refinement. Experiments on LRS3 demonstrate state-of-the-art results for AVSR. Under additive-noise conditions at 0 dB SNR, it achieves 37% relative improvement over the baseline system.
Abstract (translated)
音频-视觉语音识别（AVSR）结合了声学和视觉信息，以增强在不良声学条件下的鲁棒性。近年来，在大型语言模型（LLMs）方面的进展已经产生了具有竞争力的自动语音识别性能，并且对于AVSR也显示出有效性。然而，之前的方法独立地投影音频和视觉特征或将浅层融合应用到它们之上，这限制了跨模态对齐和互补交换，并增加了LLM的计算负担。为了解决这个问题，我们提出了AVUR-LLM，这是一种基于大型语言模型（LLMs）的语音识别方法，通过稀疏模式对齐和视觉单元引导优化来实现音频-视觉语音识别。 在LRS3数据集上的实验显示，该方法达到了最先进的AVSR结果，在0 dB信噪比下的加性噪声条件下，与基线系统相比，相对改进达到了37%。
URL
https://arxiv.org/abs/2603.03811
PDF
-
An Investigation Into Various Approaches For Bengali Long-Form Speech Transcription and Bengali Speaker Diarization
2026-03-03 17:00:42Epshita Jahan, Khandoker Md Tanjinul Islam, Pritom Biswas, Tafsir Al NafinAbstract
Bengali remains a low-resource language in speech technology, especially for complex tasks like long-form transcription and speaker diarization. This paper presents a multistage approach developed for the "DL Sprint 4.0 - Bengali Long-Form Speech Recognition" and "DL Sprint 4.0 - Bengali Speaker Diarization" competitions on Kaggle, addressing the challenge of "who spoke when/what" in hour-long recordings. We implemented Whisper Medium fine-tuned on Bengali data (bengaliAI/tugstugi bengaliai-asr whisper-medium) for transcription and integrated pyannote/speaker-diarization-community-1 with our custom-trained segmentation model to handle diverse and noisy acoustic environments. Using a two-pass method with hyperparameter tuning, we achieved a DER of 0.27 on the private leaderboard and 0.19 on the public leaderboard. For transcription, chunking, background noise cleaning, and algorithmic post-processing yielded a WER of 0.38 on the private leaderboard. These results show that targeted tuning and strategic data utilization can significantly improve AI inclusivity for South Asian languages. All relevant code is available at: this https URL Index Terms: Bengali speech recognition, speaker diarization, Whisper, ASR, low-resource languages, pyannote, voice activity detection
Abstract (translated)
孟加拉语在语音技术领域仍然是资源匮乏的语言，尤其是在长篇转录和说话人划分等复杂任务方面。本文介绍了一种多阶段方法，该方法用于Kaggle上的“DL Sprint 4.0 - 孟加拉语长文语音识别”和“DL Sprint 4.0 - 孟加拉语说话人划分”竞赛，以解决小时级录音中“谁在什么时候说了什么”的挑战。我们使用了针对孟加拉语数据微调的Whisper Medium（由bengaliAI/tugstugi bengaliai-asr whisper-medium提供）进行转录，并结合pyannote/speaker-diarization-community-1和自定义训练的分割模型来处理各种嘈杂的声学环境。通过采用两步法并调整超参数，我们在私人排行榜上实现了0.27的DER（错误接受率），在公共排行榜上实现为0.19。对于转录，我们采用了切片、背景噪音清理和算法后处理技术，在私人排行榜上达到了0.38的WER（单词错误率）。这些结果显示了针对特定语言进行针对性调整和策略性数据利用可以显著提高AI对南亚语言的包容性。所有相关代码均可在此链接访问：this https URL 关键词包括：孟加拉语语音识别、说话人划分、Whisper、ASR（自动语音识别）、资源匮乏的语言、pyannote、声音活动检测。
URL
https://arxiv.org/abs/2603.03158
PDF
-
GLoRIA: Gated Low-Rank Interpretable Adaptation for Dialectal ASR
2026-03-02 23:10:09Pouya Mehralian, Melissa Farasyn, Anne Breitbarth, Anne-Sophie Ghyselen, Hugo Van hammeAbstract
Automatic Speech Recognition (ASR) in dialect-heavy settings remains challenging due to strong regional variation and limited labeled data. We propose GLoRIA, a parameter-efficient adaptation framework that leverages metadata (e.g., coordinates) to modulate low-rank updates in a pre-trained encoder. GLoRIA injects low-rank matrices into each feed-forward layer, with a gating MLP determining the non-negative contribution of each LoRA rank-1 component based on location metadata. On the GCND corpus, GLoRIA outperforms geo-conditioned full fine-tuning, LoRA, and both dialect-specific and unified full fine-tuning, achieving state-of-the-art word error rates while updating under 10% of parameters. GLoRIA also generalizes well to unseen dialects, including in extrapolation scenarios, and enables interpretable adaptation patterns that can be visualized geospatially. These results show metadata-gated low-rank adaptation is an effective, interpretable, and efficient solution for dialectal ASR.
Abstract (translated)
在方言浓厚的环境下，自动语音识别（ASR）依然面临挑战，这主要是由于区域差异显著以及标注数据有限。为此，我们提出了一种名为GLoRIA的参数高效适应框架，该框架利用元数据（如坐标信息）来调节预训练编码器中的低秩更新。GLoRIA在每个前馈层中注入了低秩矩阵，并使用一个门控多层感知机（MLP）根据位置元数据确定每种LoRA秩-1成分的非负贡献度。 在GCND语料库上的实验表明，GLoRIA的表现优于地理条件下的全量微调、LoRA方法以及特定方言和统一全量微调方案，同时它仅更新不到10%的参数就能实现最优的话音错误率。此外，该框架还能很好地泛化到未见过的方言中，包括在推断场景下也能保持性能，并且能够生成可解释的适应模式，这些模式可以以地理空间的方式进行可视化。 实验结果表明，基于元数据门控的低秩适应是一种有效、可解释和高效的解决方案，适用于解决方言ASR问题。
URL
https://arxiv.org/abs/2603.02464
PDF
-
RO-N3WS: Enhancing Generalization in Low-Resource ASR with Diverse Romanian Speech Benchmarks
2026-03-02 20:14:31Alexandra Diaconu, M\u{a}d\u{a}lina V\^inaga, Bogdan AlexeAbstract
We introduce RO-N3WS, a benchmark Romanian speech dataset designed to improve generalization in automatic speech recognition (ASR), particularly in low-resource and out-of-distribution (OOD) conditions. RO-N3WS comprises over 126 hours of transcribed audio collected from broadcast news, literary audiobooks, film dialogue, children's stories, and conversational podcast speech. This diversity enables robust training and fine-tuning across stylistically distinct domains. We evaluate several state-of-the-art ASR systems (Whisper, Wav2Vec 2.0) in both zero-shot and fine-tuned settings, and conduct controlled comparisons using synthetic data generated with expressive TTS models. Our results show that even limited fine-tuning on real speech from RO-N3WS yields substantial WER improvements over zero-shot baselines. We will release all models, scripts, and data splits to support reproducible research in multilingual ASR, domain adaptation, and lightweight deployment.
Abstract (translated)
我们介绍了RO-N3WS，这是一个为改善自动语音识别（ASR）系统在低资源和分布外（OOD）条件下的泛化能力而设计的罗马尼亚语语音数据集。RO-N3WS包含超过126小时经过转录的音频资料，这些资料来自广播新闻、文学有声书、电影对话、儿童故事以及对话型播客演讲等不同来源。这种多样性使得在风格迥异的领域内进行稳健的训练和微调成为可能。 我们在零样本和微调两种设置下评估了几种最先进的ASR系统（如Whisper和Wav2Vec 2.0），并使用具有表现力的TTS模型生成的合成数据进行了控制比较。我们的结果显示，即使是在RO-N3WS的真实语音上进行有限度的微调也能显著改善相对于零样本基准条件下的错误词率（WER）。 为了支持多语言ASR、领域适应和轻量级部署方面的可重复研究工作，我们将发布所有模型、脚本及数据分割。
URL
https://arxiv.org/abs/2603.02368
PDF
-
VietSuperSpeech: A Large-Scale Vietnamese Conversational Speech Dataset for ASR Fine-Tuning in Chatbot, Customer Support, and Call Center Applications
2026-03-02 14:13:57Loan Do, Thanh Ngoc Nguyen, Thanh Pham, Vinh Do, Hien Nguyen, Charlotte NguyenAbstract
We introduce VietSuperSpeech, a large-scale Vietnamese automatic speech recognition (ASR) dataset of 52,023 audio-text pairs totaling 267.39 hours, with a distinctive focus on casual conversational speech. Unlike existing Vietnamese ASR corpora that predominantly feature read speech, news narration, or audiobook content, VietSuperSpeech is sourced from four publicly accessible YouTube channels spanning everyday conversation, personal vlogging, overseas Vietnamese community dialogue, and informal commentary - the very speech styles encountered in real-world chatbot, customer support, call center, and hotline deployments. All audio is standardized to 16 kHz mono PCM WAV and segmented into 3-30 second utterances. Transcriptions are generated via pseudo-labeling using the Zipformer-30M-RNNT-6000h model (Nguyen, 2025) deployed through Sherpa-ONNX, pre-trained on 6,000 hours of Vietnamese speech. After quality filtering, the dataset is split into 46,822 training samples (240.67 hours) and 5,201 development/test samples (26.72 hours) with a fixed random seed. The text averages 266 characters per utterance, totaling 13.8 million fully diacritically marked Vietnamese characters. We demonstrate that VietSuperSpeech fills a critical gap in the Vietnamese ASR ecosystem: while corpora such as VLSP2020, VIET_BUD500, VietSpeech, FLEURS, VietMed, Sub-GigaSpeech2-Vi, viVoice, and Sub-PhoAudioBook provide broad coverage of formal and read speech, none specifically targets the casual, spontaneous register indispensable for conversational AI applications. VietSuperSpeech is publicly released at this https URL.
Abstract (translated)
我们介绍了VietSuperSpeech，这是一个包含52,023对音频文本数据、总计267.39小时的大型越南语自动语音识别（ASR）数据集，特别侧重于日常对话中的非正式会话内容。与现有主要收录朗读、新闻播报或有声书内容的越南语ASR语料库不同，VietSuperSpeech的数据来源于四个公开访问的YouTube频道，这些渠道涵盖了日常生活对话、个人博客视频、海外越南社区对话以及非正式评论——这些都是实际应用中聊天机器人、客户支持和呼叫中心会遇到的真实语音风格。所有音频统一标准化为16 kHz单声道PCM WAV格式，并被分割成3到30秒长的片段。 转录文本是通过使用Zipformer-30M-RNNT-6000h模型（Nguyen, 2025）进行伪标签生成，该模型在经过6,000小时越南语语音预训练后部署于Sherpa-ONNX平台。质量过滤之后，数据集被分为46,822个用于训练的样本（总计240.67小时）和5,201个开发/测试样本（共计26.72小时），并采用固定随机种子进行划分。每个语音片段的文本平均长度为266字符，整个数据集包含约1380万完全带音标符号的越南语字符。 我们展示了VietSuperSpeech填补了越南语ASR生态系统中的关键空白：虽然诸如VLSP2020、VIET_BUD500、VietSpeech、FLEURS、VietMed、Sub-GigaSpeech2-Vi、viVoice和Sub-PhoAudioBook等数据集广泛涵盖了正式及朗读语音，但没有一个专门针对非正式、即兴的日常会话场景，而这对于对话型AI应用至关重要。VietSuperSpeech可以在[这里](https://this-url.com)公开获取（原文链接处应填写具体网址）。
URL
https://arxiv.org/abs/2603.01894
PDF
-
From Black Box to Glass Box: Cross-Model ASR Disagreement to Prioto Review in Ambient AI Scribe Documentation
2026-03-02 13:02:13Abdolamir Karbalaie, Fernando Seoane, Farhad AbtahiAbstract
Ambient AI "scribe" systems promise to reduce clinical documentation burden, but automatic speech recognition (ASR) errors can remain unnoticed without careful review, and high-quality human reference transcripts are often unavailable for calibrating uncertainty. We investigate whether cross-model disagreement among heterogeneous ASR systems can act as a reference-free uncertainty signal to prioritize human verification in medical transcription workflows. Using 50 publicly available medical education audio clips (8 h 14 min), we transcribed each clip with eight ASR systems spanning commercial APIs and open-source engines. We aligned multi-model outputs, built consensus pseudo-references, and quantified token-level agreement using a majority-strength metric; we further characterized disagreements by type (content vs. punctuation/formatting) and assessed per-model agreement via leave-one-model-out (jackknife) consensus scoring. Inter-model reliability was low (ICC[2,1] = 0.131), indicating heterogeneous failure modes across systems. Across 76,398 evaluated token positions, 72.1% showed near-unanimous agreement (7-8 models), while 2.5% fell into high-risk bands (0-3 models), with high-risk mass varying from 0.7% to 11.4% across accent groups. Low-agreement regions were enriched for content disagreements, with the content fraction increasing from 53.9% to 73.9% across quintiles of high-risk mass. These results suggest that cross-model disagreement provides a sparse, localizable signal that can surface potentially unreliable transcript spans without human-verified references, enabling targeted review; clinical accuracy of flagged regions remains to be established.
Abstract (translated)
URL
https://arxiv.org/abs/2604.14152
PDF
-
Sequence-Level Unsupervised Training in Speech Recognition: A Theoretical Study
2026-03-02 11:09:17Zijian Yang, J\"org Barkoczi, Ralf Schl\"uter, Hermann NeyAbstract
Unsupervised speech recognition is a task of training a speech recognition model with unpaired data. To determine when and how unsupervised speech recognition can succeed, and how classification error relates to candidate training objectives, we develop a theoretical framework for unsupervised speech recognition grounded in classification error bounds. We introduce two conditions under which unsupervised speech recognition is possible. The necessity of these conditions are also discussed. Under these conditions, we derive a classification error bound for unsupervised speech recognition and validate this bound in simulations. Motivated by this bound, we propose a single-stage sequence-level cross-entropy loss for unsupervised speech recognition.
Abstract (translated)
无监督语音识别是指使用未配对数据训练语音识别模型的任务。为了确定何时以及如何能够成功进行无监督语音识别，以及分类误差与候选训练目标之间的关系，我们基于分类误差界限建立了一个理论框架来研究无监督语音识别。我们引入了两个在特定情况下可行的条件，并探讨了这些条件的必要性。在此基础上，我们在这些条件下推导出了无监督语音识别的分类误差界，并通过模拟验证了这一边界。受到这个界的启发，我们提出了一种用于无监督语音识别的一阶段序列级交叉熵损失方法。
URL
https://arxiv.org/abs/2603.02285
PDF
-
More Data, Fewer Diacritics: Scaling Arabic TTS
2026-03-02 08:58:50Ahmed Musleh, Yifan Zhang, Kareem DarwishAbstract
Arabic Text-to-Speech (TTS) research has been hindered by the availability of both publicly available training data and accurate Arabic diacritization models. In this paper, we address the limitation by exploring Arabic TTS training on large automatically annotated data. Namely, we built a robust pipeline for collecting Arabic recordings and processing them automatically using voice activity detection, speech recognition, automatic diacritization, and noise filtering, resulting in around 4,000 hours of Arabic TTS training data. We then trained several robust TTS models with voice cloning using varying amounts of data, namely 100, 1,000, and 4,000 hours with and without diacritization. We show that though models trained on diacritized data are generally better, larger amounts of training data compensate for the lack of diacritics to a significant degree. We plan to release a public Arabic TTS model that works without the need for diacritization.
Abstract (translated)
阿拉伯语文本到语音（TTS）的研究一直受到公开可用的训练数据和准确的阿拉伯语标点模型稀缺性的限制。在本文中，我们通过探索使用大量自动注释的数据进行阿拉伯语TTS训练来解决这一问题。具体来说，我们建立了一个稳健的管道，用于收集阿拉伯语录音，并利用语音活动检测、语音识别、自动标音和噪声过滤等技术对其进行自动处理，最终产生了约4,000小时的阿拉伯语TTS训练数据。然后，我们在不同数量的数据（即100小时、1,000小时和4,000小时）上使用声纹克隆技术训练了几个稳健的TTS模型，并分别在有标点和无标点的情况下进行了测试。我们发现，虽然基于标点数据训练出来的模型通常表现更好，但大量的训练数据可以在很大程度上弥补缺少标音的问题。我们的计划是发布一个无需依赖标音即可工作的公共阿拉伯语TTS模型。
URL
https://arxiv.org/abs/2603.01622
PDF
-
End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation
2026-03-02 02:26:01Minghui Wu, Haitao Tang, Jiahuan Fan, Ruizhi Liao, Yanyong ZhangAbstract
Dysarthric speech reconstruction (DSR) typically employs a cascaded system that combines automatic speech recognition (ASR) and sentence-level text-to-speech (TTS) to convert dysarthric speech into normally-prosodied speech. However, dysarthric individuals often speak more slowly, leading to excessively long response times in such systems, rendering them impractical in long-speech scenarios. Cascaded DSR systems based on streaming ASR and incremental TTS can help reduce latency. However, patients with differing dysarthria severity exhibit substantial pronunciation variability for the same text, resulting in poor robustness of ASR and limiting the intelligibility of reconstructed speech. In addition, incremental TTS suffers from poor prosodic feature prediction due to a limited receptive field. In this study, we propose an end-to-end simultaneous DSR system with two key innovations: 1) A frame-level adaptor module is introduced to bridge ASR and TTS. By employing explicit-implicit semantic information fusion and joint module training, it enhances the error tolerance of TTS to ASR outputs. 2) A multiple wait-k autoregressive TTS module is designed to mitigate prosodic degradation via multi-view knowledge distillation. Our system has an average response time of 1.03 seconds on Tesla A100, with an average real-time factor (RTF) of 0.71. On the UASpeech dataset, it attains a mean opinion score (MOS) of 4.67 and demonstrates a 54.25% relative reduction in word error rate (WER) compared to the state-of-the-art. Our demo is available at: this https URL
Abstract (translated)
失语症语音重建（DSR）通常采用级联系统，结合自动语音识别（ASR）和句子级别的文本到语音合成（TTS），将失语症患者的语音转换为正常的音调语音。然而，失语症患者往往说话速度较慢，导致此类系统的响应时间过长，在处理长时间对话时显得不切实际。基于流式ASR和增量TTS的级联DSR系统有助于减少延迟。但是，患有不同程度失语症的病人对于同一文本发音差异显著，这会导致ASR的鲁棒性较差，并限制了重建语音的可理解性。此外，由于感受野有限，增量TTS在预测音调特征方面表现不佳。 在这项研究中，我们提出了一种端到端的同时DSR系统，具有两个关键创新：1）引入了一个帧级适配器模块来连接ASR和TTS。通过采用显式-隐式语义信息融合及联合模块训练，它增强了TTS对ASR输出的错误容忍度；2）设计了一种多等待-k自回归TTS模块，通过多视角知识蒸馏缓解音调退化问题。 我们的系统在Tesla A100上的平均响应时间为1.03秒，实时因子（RTF）平均为0.71。在UASpeech数据集上，它获得了4.67的平均意见得分（MOS），并且相较于最先进的方法，单词错误率（WER）相对降低了54.25%。 我们的演示可在以下链接查看：[此处提供链接]
URL
https://arxiv.org/abs/2603.01382
PDF
-
DARS: Dysarthria-Aware Rhythm-Style Synthesis for ASR Enhancement
2026-03-02 02:05:14Minghui Wu, Xueling Liu, Jiahuan Fan, Haitao Tang, Yanyong Zhang, Yue ZhangAbstract
Dysarthric speech exhibits abnormal prosody and significant speaker variability, presenting persistent challenges for automatic speech recognition (ASR). While text-to-speech (TTS)-based data augmentation has shown potential, existing methods often fail to accurately model the pathological rhythm and acoustic style of dysarthric speech. To address this, we propose DARS, a dysarthria-aware rhythm-style synthesis framework based on the Matcha-TTS architecture. DARS incorporates a multi-stage rhythm predictor optimized by contrastive preferences between normal and dysarthric speech, along with a dysarthric-style conditional flow matching mechanism, jointly enhancing temporal rhythm reconstruction and pathological acoustic style simulation. Experiments on the TORGO dataset demonstrate that DARS achieves a Mean Cepstral Distortion (MCD) of 4.29, closely approximating real dysarthric speech. Adapting a Whisper-based ASR system with synthetic dysarthric speech from DARS achieves a 54.22% relative reduction in word error rate (WER) compared to state-of-the-art methods, demonstrating the framework's effectiveness in enhancing recognition performance.
Abstract (translated)
构音障碍语音表现出异常的语调和显著的说话人变异，这对自动语音识别（ASR）构成了持续挑战。虽然基于文本到语音（TTS）的数据增强方法显示出潜力，但现有方法往往无法准确模拟构音障碍语音的病理节奏和声学风格。为了解决这一问题，我们提出了DARS框架——一个基于Matcha-TTS架构、针对构音障碍语音感知的节律-风格合成框架。DARS整合了多阶段的节奏预测器，该预测器通过正常和构音障碍话语之间的对比偏好进行优化，并且包含了一个模拟构音障碍声学风格的条件流匹配机制，共同增强了时间节奏重建和病理声学风格模拟的能力。 在TORGO数据集上的实验表明，DARS实现了4.29的平均倒谱失真（MCD），与实际构音障碍语音非常接近。当使用DARS生成的合成构音障碍话语来适应基于Whisper的ASR系统时，该系统的词错误率（WER）相比现有最佳方法降低了54.22%，这证明了该框架在提高识别性能方面的有效性。
URL
https://arxiv.org/abs/2603.01369
PDF
-
Towards Orthographically-Informed Evaluation of Speech Recognition Systems for Indian Languages
2026-03-01 06:20:35Kaushal Santosh Bhogale, Tahir Javed, Greeshma Susan John, Dhruv Rathi, Akshayasree Padmanaban, Niharika Parasa, Mitesh M. KhapraAbstract
Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Word Error Rate (WER) often presents a bleaker picture of system performance than what human users perceive. Better aligning evaluation with real-world performance requires capturing permissible orthographic variations, which is extremely challenging for under-resourced Indian languages. Leveraging recent advances in LLMs, we propose a framework for creating benchmarks that capture permissible variations. Through extensive experiments, we demonstrate that OIWER, by accounting for orthographic variations, reduces pessimistic error rates (an average improvement of 6.3 points), narrows inflated model gaps (e.g., Gemini-Canary performance difference drops from 18.1 to 11.5 points), and aligns more closely with human perception than prior methods like WER-SN by 4.9 points.
Abstract (translated)
评估印度语言的自动语音识别（ASR）系统具有挑战性，原因包括拼写变化、词尾分割灵活性以及代码混合单词中的非标准拼写。传统的词错误率（WER）常常比人类用户的实际体验更为悲观地描绘系统的性能。为了更好地将评估与现实世界的性能相匹配，需要捕捉允许的正字法变化，这对资源匮乏的印度语言来说尤其具有挑战性。借助最近在大型语言模型（LLM）领域的进展，我们提出了一种框架来创建能够捕捉允许的变化的基准测试。通过广泛的实验，我们证明了考虑到正字法变化的对象初始化词错误率（OIWER），减少了悲观的误差率（平均提高了6.3个百分点）、缩小了夸大后的模型差距（例如，Gemini-Canary性能差异从18.1降至11.5个百分点）并且比之前的评估方法如WER-SN更紧密地与人类感知对齐（高出4.9个百分点）。
URL
https://arxiv.org/abs/2603.00941
PDF
-
Polynomial Mixing for Efficient Self-supervised Speech Encoders
2026-02-28 14:45:55Eva Feillet, Ryan Whetten, David Picard, Alexandre AllauzenAbstract
State-of-the-art speech-to-text models typically employ Transformer-based encoders that model token dependencies via self-attention mechanisms. However, the quadratic complexity of self-attention in both memory and computation imposes significant constraints on scalability. In this work, we propose a novel token-mixing mechanism, the Polynomial Mixer (PoM), as a drop-in replacement for multi-head self-attention. PoM computes a polynomial representation of the input with linear complexity with respect to the input sequence length. We integrate PoM into a self-supervised speech representation learning framework based on BEST-RQ and evaluate its performance on downstream speech recognition tasks. Experimental results demonstrate that PoM achieves a competitive word error rate compared to full self-attention and other linear-complexity alternatives, offering an improved trade-off between performance and efficiency in time and memory.
Abstract (translated)
最先进的语音转文本模型通常采用基于Transformer的编码器，通过自注意力机制来建模标记依赖关系。然而，自注意力在内存和计算上的二次复杂度对可扩展性带来了显著限制。在这项工作中，我们提出了一种新颖的标记混合机制——多项式混合器（Polynomial Mixer, PoM），作为多头自注意力的即插即用替代方案。PoM以线性复杂度计算输入的多项式表示，相对于输入序列长度而言。我们将PoM整合进一个基于BEST-RQ的自监督语音表征学习框架中，并对其在下游语音识别任务上的性能进行评估。实验结果表明，PoM在词错误率上与完整的自注意力和其他线性复杂度替代方案相当，但在时间和内存效率方面提供了更好的性能-效率平衡。
URL
https://arxiv.org/abs/2603.00683
PDF
-
Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion
2026-02-28 09:24:01Sen Zhang, Jianguo Wei, Wenhuan Lu, Xianghu Yue, Wei Li, Qiang Li, Pengcheng Zhao, Ming Cai, Luo SiAbstract
The Transformer-based Whisper model has achieved state-of-the-art performance in Automatic Speech Recognition (ASR). However, its Multi-Head Attention (MHA) mechanism results in significant GPU memory consumption due to the linearly growing Key-Value (KV) cache usage, which is problematic for many applications especially with long-form audio. To address this, we introduce Whisper-MLA, a novel architecture that incorporates Multi-Head Latent Attention (MLA) into the Whisper model. Specifically, we adapt MLA for Whisper's absolute positional embeddings and systematically investigate its application across encoder self-attention, decoder self-attention, and cross-attention modules. Empirical results indicate that applying MLA exclusively to decoder self-attention yields the desired balance between performance and memory efficiency. Our proposed approach allows conversion of a pretrained Whisper model to Whisper-MLA with minimal fine-tuning. Extensive experiments on the LibriSpeech benchmark validate the effectiveness of this conversion, demonstrating that Whisper-MLA reduces the KV cache size by up to 87.5% while maintaining competitive accuracy.
Abstract (translated)
基于Transformer的Whisper模型在自动语音识别（ASR）方面取得了最先进的性能。然而，其多头注意力（MHA）机制由于键值（KV）缓存使用量线性增长而导致显著的GPU内存消耗，这对许多应用尤其是长音频处理来说是一个问题。为解决这一挑战，我们引入了Whisper-MLA，这是一种将多头潜在注意力（MLA）集成到Whisper模型中的新型架构。具体而言，我们将MLA适应于Whisper的绝对位置嵌入，并系统地研究其在编码器自注意力、解码器自注意力和交叉注意力模块中的应用。实证结果表明，仅将MLA应用于解码器自注意力即可达到性能与内存效率之间的理想平衡。 我们提出的转换方法允许将预训练的Whisper模型最小化微调后转化为Whisper-MLA。在LibriSpeech基准数据集上的大量实验验证了这一转换的有效性，证明了Whisper-MLA可以减少KV缓存大小高达87.5%，同时保持竞争力的准确性。
URL
https://arxiv.org/abs/2603.00563
PDF
-
ACES: Accent Subspaces for Coupling, Explanations, and Stress-Testing in Automatic Speech Recognition
Abstract
ASR systems exhibit persistent performance disparities across accents, yet the internal mechanisms underlying these gaps remain poorly understood. We introduce ACES, a representation-centric audit that extracts accent-discriminative subspaces and uses them to probe model fragility and disparity. Analyzing Wav2Vec2-base with five English accents, we find that accent information concentrates in a low-dimensional early-layer subspace (layer 3, k=8). Projection magnitude correlates with per-utterance WER (r=0.26), and crucially, subspace-constrained perturbations yield stronger coupling between representation shift and degradation (r=0.32) than random-subspace controls (r=0.15). Finally, linear attenuation of this subspace however does not reduce disparity and slightly worsens it. Our findings suggest that accent-relevant features are deeply entangled with recognition-critical cues, positioning accent subspaces as vital diagnostic tools rather than simple "erasure" levers for fairness.
Abstract (translated)
ASR系统在不同口音之间的性能差异持续存在，但导致这些差距的内部机制仍然不甚明了。我们引入了一种名为ACES的表示中心审计方法，该方法提取出具有口音区分性的子空间，并利用它们来探测模型的脆弱性和差异性。通过分析Wav2Vec2-base在五种英语口音上的表现，我们发现口音信息集中在低维早期层（第三层，k=8）的空间中。投影幅度与每次发音的WER（单词错误率）相关联（r=0.26），更重要的是，在子空间约束下的扰动产生了更强烈的表示变化和性能下降之间的耦合（r=0.32），这比随机选择子空间控制实验的结果（r=0.15）更为显著。最后，线性衰减该子空间并没有减少差异，反而稍微加剧了这种差异。我们的研究结果表明，与识别关键线索紧密相连的口音相关特征是深深交织在一起的，因此将这些口音子空间视为重要的诊断工具而非简单的“消除”手段对于实现公平性而言更为重要。
URL
https://arxiv.org/abs/2603.03359
PDF
-
Dialect and Gender Bias in YouTube's Spanish Captioning System
2026-02-27 13:26:42Iris Dania Jimenez, Christoph KernAbstract
Spanish is the official language of twenty-one countries and is spoken by over 441 million people. Naturally, there are many variations in how Spanish is spoken across these countries. Media platforms such as YouTube rely on automatic speech recognition systems to make their content accessible to different groups of users. However, YouTube offers only one option for automatically generating captions in Spanish. This raises the question: could this captioning system be biased against certain Spanish dialects? This study examines the potential biases in YouTube's automatic captioning system by analyzing its performance across various Spanish dialects. By comparing the quality of captions for female and male speakers from different regions, we identify systematic disparities which can be attributed to specific dialects. Our study provides further evidence that algorithmic technologies deployed on digital platforms need to be calibrated to the diverse needs and experiences of their user populations.
Abstract (translated)
西班牙是二十一个国家的官方语言，全球有超过4亿4100万人使用这种语言。然而，在这些国家中，西班牙语的口音和用法存在许多差异。像YouTube这样的媒体平台依赖自动语音识别系统来为其用户提供可访问的内容。但是，YouTube仅提供一种自动生成西班牙语字幕的选择。这引发了这样一个问题：该字幕生成系统是否会对某些西班牙语方言产生偏见？本研究通过分析YouTube的自动字幕系统在不同西班牙语方言中的表现，探讨其潜在的偏见。我们通过对来自不同地区的女性和男性的演讲者产生的字幕质量进行比较，识别出可以归因于特定方言的系统性差异。我们的研究表明，在线平台部署的算法技术需要针对用户群体多样化的需要和体验进行校准。
URL
https://arxiv.org/abs/2602.24002
PDF
-
Whisper-RIR-Mega: A Paired Clean-Reverberant Speech Benchmark for ASR Robustness to Room Acoustics
Abstract
We introduce Whisper-RIR-Mega, a benchmark dataset of paired clean and reverberant speech for evaluating automatic speech recognition (ASR) robustness to room acoustics. Each sample pairs a clean LibriSpeech utterance with the same utterance convolved with a real room impulse response from the RIR-Mega corpus, with stratified splits by reverberation time (RT60) and direct-to-reverberant ratio (DRR). We evaluate five Whisper models (tiny through large-v3) on 1600 test samples and report word error rate (WER) and character error rate (CER) under clean and reverberant conditions. Reverberation consistently degrades performance across all model sizes; the reverb penalty in WER ranges from 0.12 to 1.07 percentage points depending on the model. We release the dataset, evaluation code, and baseline results to support reproducible research on robust ASR.
Abstract (translated)
我们介绍了Whisper-RIR-Mega，这是一个用于评估自动语音识别（ASR）在房间声学条件下鲁棒性的配对纯净和回响语音基准数据集。每个样本包含一个来自LibriSpeech的纯净发音与该发音卷积后的实际房间脉冲响应（从RIR-Mega语料库中获取），并按照混响时间（RT60）和直达声到反射声比率（DRR）进行了分层划分。我们在1600个测试样本上评估了五个Whisper模型（tiny至large-v3），并在纯净和回响条件下报告单词错误率（WER）和字符错误率（CER）。混响效应在所有模型大小中都导致性能下降；根据模型的不同，WER的混响惩罚范围从0.12到1.07个百分点。我们发布了该数据集、评估代码及基线结果，以支持稳健ASR研究的可重复性。
URL
https://arxiv.org/abs/2603.02252
PDF
-
Breeze Taigi: Benchmarks and Models for Taiwanese Hokkien Speech Recognition and Synthesis
2026-02-26 16:35:16Yu-Siang Lan, Chia-Sheng Liu, Yi-Chang Chen, Po-Chun Hsu, Allyson Chiu, Shun-Wen Lin, Da-shan Shiu, Yuan-Fu LiaoAbstract
Taiwanese Hokkien (Taigi) presents unique opportunities for advancing speech technology methodologies that can generalize to diverse linguistic contexts. We introduce Breeze Taigi, a comprehensive framework centered on standardized benchmarks for evaluating Taigi speech recognition and synthesis systems. Our primary contribution is a reproducible evaluation methodology that leverages parallel Taiwanese Mandarin resources. We provide 30 carefully curated Mandarin-Taigi audio pairs from Taiwan's Executive Yuan public service announcements with normalized ground truth transcriptions. We establish Character Error Rate (CER) as the standard metric and implement normalization procedures to enable fair cross-system comparisons. To demonstrate the benchmark's utility and provide reference implementations, we develop speech recognition and synthesis models through a methodology that leverages existing Taiwanese Mandarin resources and large-scale synthetic data generation. In particular, we fine-tune a Whisper model on approximately 10,000 hours of Taigi synthetic speech data. Our ASR model achieves 30.13% average CER on the benchmark, outperforming existing commercial and research systems. By providing standardized evaluation protocols, diverse training datasets, and open baseline models, we offer a replicable framework with methodologies applicable to various linguistic contexts.
Abstract (translated)
台语（Taigi）为推进可泛化至多元语言场景的语音技术方法论提供了独特机遇。本文提出Breeze Taigi框架，该框架以标准化基准测试为核心，用于评估台语语音识别与合成系统。我们的核心贡献是建立了一套可复现的评估方法论，充分利用平行台语-普通话资源。我们提供30对经精心筛选的普通话-台语音频样本，源自台湾行政院公共事务 announcements，并配备标准化真值转写。我们确立字错误率（CER）作为标准评估指标，并实施标准化流程以实现跨系统公平对比。为验证基准实用性并提供参考实现，我们通过利用现有台语-普通话资源及大规模合成数据生成的方法论，开发了语音识别与合成模型。特别地，我们基于约10,000小时台语合成语音数据对Whisper模型进行了微调。我们的ASR模型在基准测试中取得30.13%的平均CER，性能超越现有商业及研究系统。通过提供标准化评估协议、多样化训练数据集及开源基线模型，我们构建了一套可复现的框架，其方法论适用于各类语言场景。
URL
https://arxiv.org/abs/2603.19259
PDF
-
Make It Hard to Hear, Easy to Learn: Long-Form Bengali ASR and Speaker Diarization via Extreme Augmentation and Perfect Alignment
2026-02-26 14:59:24Sanjid Hasan, Risalat Labib, A H M Fuad, Bayazid HasanAbstract
Although Automatic Speech Recognition (ASR) in Bengali has seen significant progress, processing long-duration audio and performing robust speaker diarization remain critical research gaps. To address the severe scarcity of joint ASR and diarization resources for this language, we introduce Lipi-Ghor-882, a comprehensive 882-hour multi-speaker Bengali dataset. In this paper, detailing our submission to the DL Sprint 4.0 competition, we systematically evaluate various architectures and approaches for long-form Bengali speech. For ASR, we demonstrate that raw data scaling is ineffective; instead, targeted fine-tuning utilizing perfectly aligned annotations paired with synthetic acoustic degradation (noise and reverberation) emerges as the singular most effective approach. Conversely, for speaker diarization, we observed that global open-source state-of-the-art models (such as Diarizen) performed surprisingly poorly on this complex dataset. Extensive model retraining yielded negligible improvements; instead, strategic, heuristic post-processing of baseline model outputs proved to be the primary driver for increasing accuracy. Ultimately, this work outlines a highly optimized dual pipeline achieving a $\sim$0.019 Real-Time Factor (RTF), establishing a practical, empirically backed benchmark for low-resource, long-form speech processing.
Abstract (translated)
尽管孟加拉语的自动语音识别（ASR）取得了显著进展，但处理长时间音频和进行稳健的说话人定位仍然是重要的研究空白。为了解决这种语言在联合ASR和说话人定位资源上的严重不足，我们引入了Lipi-Ghor-882，这是一个全面的882小时多说话者孟加拉语数据集。在这篇论文中，详细介绍了我们在DL Sprint 4.0竞赛中的提交内容，系统地评估了各种长格式孟加拉语音架构和方法。 对于ASR，我们证明了单纯的数据扩充效果不佳；相反，采用完美对齐的注释，并结合合成声学退化（如噪音和回响）进行目标细化是唯一最有效的策略。相比之下，在说话人定位方面，全球开源最先进的模型（如Diarizen），在处理这个复杂数据集时表现得出乎意料地差。大量的重新训练并未带来显著改进；反而，基于基线模型输出的战略性、启发式的后处理成为提高准确性的主要驱动力。 最终，这项工作概述了一种高度优化的双管道方法，达到了约0.019的真实时间因子（RTF），为低资源长时间语音处理建立了一个实用且经验支持的基准。
URL
https://arxiv.org/abs/2602.23070
PDF
-
A Holistic Framework for Robust Bangla ASR and Speaker Diarization with Optimized VAD and CTC Alignment
2026-02-26 12:26:04Zarif Ishmam, Zarif Mahir, Shafnan Wasif, Md. Ishtiak MoinAbstract
Despite being one of the most widely spoken languages globally, Bangla remains a low-resource language in the field of Natural Language Processing (NLP). Mainstream Automatic Speech Recognition (ASR) and Speaker Diarization systems for Bangla struggles when processing longform audio exceeding 3060 seconds. This paper presents a robust framework specifically engineered for extended Bangla content by leveraging preexisting models enhanced with novel optimization pipelines for the DL Sprint 4.0 contest. Our approach utilizes Voice Activity Detection (VAD) optimization and Connectionist Temporal Classification (CTC) segmentation via forced word alignment to maintain temporal accuracy and transcription integrity over long durations. Additionally, we employed several finetuning techniques and preprocessed the data using augmentation techniques and noise removal. By bridging the performance gap in complex, multi-speaker environments, this work provides a scalable solution for real-world, longform Bangla speech applications.
Abstract (translated)
尽管孟加拉语是全球使用最广泛的语言之一，但在自然语言处理（NLP）领域中，它仍然是一种资源匮乏的语言。主流的自动语音识别（ASR）和说话人聚类系统在处理超过3060秒的长音频时会遇到困难。本文提出了一种专门针对扩展孟加拉语内容设计的强大框架，并通过创新优化管道增强了现有的模型，以适应DL Sprint 4.0竞赛的要求。 我们的方法利用了语音活动检测（VAD）优化和连接时间分类（CTC）分段技术，通过强制词对齐来保持长时间内的时序准确性和转录完整性。此外，我们还采用了多种微调技术和数据预处理措施，包括使用增强技术和噪声消除来提高模型性能。 通过解决复杂多说话人环境中的性能差距问题，本研究为实际长音频孟加拉语应用提供了一种可扩展的解决方案。
URL
https://arxiv.org/abs/2602.22935
PDF
