Speech_Recognition
-
SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases
2026-03-10 16:15:12Laya Iyer, Angelina Wang, Sanmi KoyejoAbstract
Advances in large language models (LLMs) have enabled significant capabilities in audio processing, resulting in state-of-the-art models now known as Large Audio Language Models (LALMs). However, minimal work has been done to measure audio understanding beyond automatic speech recognition (ASR). This paper closes that gap by proposing a benchmark suite, SCENEBench (Spatial, Cross-lingual, Environmental, Non-speech Evaluation), that targets a broad form of audio comprehension across four real-world categories: background sound understanding, noise localization, cross-linguistic speech understanding, and vocal characterizer recognition. These four categories are selected based on understudied needs from accessibility technology and industrial noise monitoring. In addition to performance, we also measure model latency. The purpose of this benchmark suite is to assess audio beyond just what words are said - rather, how they are said and the non-speech components of the audio. Because our audio samples are synthetically constructed (e.g., by overlaying two natural audio samples), we further validate our benchmark against 20 natural audio items per task, sub-sampled from existing datasets to match our task criteria, to assess ecological validity. We assess five state-of-the-art LALMs and find critical gaps: performance varies across tasks, with some tasks performing below random chance and others achieving high accuracy. These results provide direction for targeted improvements in model capabilities.
Abstract (translated)
大型语言模型（LLM）的进步使音频处理能力得到了显著提升，进而产生了被称为大音频语言模型（LALM）的先进模型。然而，在自动语音识别（ASR）之外对音频理解进行度量的工作仍然很少。本文通过提出一个基准测试套件SCENEBench来填补这一空白，该套件涵盖了四大类现实世界中的音频理解：背景声音理解、噪音定位、跨语言口语理解和声源特征识别。这四个类别是基于无障碍技术和工业噪声监测领域中研究不足的需求而选定的。 除了性能之外，我们还对模型延迟进行了测量。此基准测试套件旨在评估不仅仅是说出了什么词，更重要的是如何说出这些话以及音频中的非言语部分。由于我们的音频样本是由合成方法构建的（例如，在自然音频样本上叠加），为了验证生态有效性，我们在每个任务中选取了20个从现有数据集中随机抽样的自然音频样本来与合成测试进行对比。 我们评估了五个最先进的LALM，并发现了重要的差距：在各个任务中的性能表现不一，有些任务的表现甚至低于随机猜测的水平，而其他任务则能达到较高的准确性。这些结果为模型能力的具体改进指明了方向。
URL
https://arxiv.org/abs/2603.09853
PDF
-
Trade-offs Between Capacity and Robustness in Neural Audio Codecs for Adversarially Robust Speech Recognition
2026-03-10 00:10:23Jordan Prescott, Thanathai Lertpetchpun, Shrikanth NarayananAbstract
Adversarial perturbations exploit vulnerabilities in automatic speech recognition (ASR) systems while preserving human perceived linguistic content. Neural audio codecs impose a discrete bottleneck that can suppress fine-grained signal variations associated with adversarial noise. We examine how the granularity of this bottleneck, controlled by residual vector quantization (RVQ) depth, shapes adversarial robustness. We observe a non-monotonic trade-off under gradient-based attacks: shallow quantization suppresses adversarial perturbations but degrades speech content, while deeper quantization preserves both content and perturbations. Intermediate depths balance these effects and minimize transcription error. We further show that adversarially induced changes in discrete codebook tokens strongly correlate with transcription error. These gains persist under adaptive attacks, where neural codec configurations outperform traditional compression defenses.
Abstract (translated)
对抗性扰动在利用自动语音识别（ASR）系统的漏洞的同时，保持人类感知的语言内容不变。神经音频编解码器施加了一个离散瓶颈，可以抑制与对抗噪声相关的细微信号变化。我们研究了这个瓶颈的粒度如何通过残差向量量化（RVQ）深度来控制，并探讨这对对抗鲁棒性的影响。我们在基于梯度的攻击下观察到一种非单调权衡：浅层量化可以抑制对抗性扰动，但会降低语音内容的质量；而深层量化则既能保留内容又能保留扰动。中间深度则能平衡这些效果并最小化转录错误。此外，我们还发现由对抗攻击引起的离散代码本令牌的变化与转录误差有很强的相关性。这种优势在自适应攻击下依然存在，其中神经编解码器配置优于传统的压缩防御措施。
URL
https://arxiv.org/abs/2603.09034
PDF
-
Dynin-Omni: Omnimodal Unified Large Diffusion Language Model
2026-03-09 13:59:02Jaeik Kim, Woojin Kim, Jihwan Hong, Yejoon Lee, Sieun Hyeon, Mintaek Lim, Yunseok Han, Dogeun Kim, Hoeun Lee, Hyunggeun Kim, Jaeyoung DoAbstract
We present Dynin-Omni, the first masked-diffusion-based omnimodal foundation model that unifies text, image, and speech understanding and generation, together with video understanding, within a single architecture. Unlike autoregressive unified models that serialize heterogeneous modalities, or compositional unified models that require orchestration with external modality-specific decoders, Dynin-Omni natively formulates omnimodal modeling as masked diffusion over a shared discrete token space, enabling iterative refinement under bidirectional context. Dynin-Omni adopts a multi-stage training strategy with model-merging-based modality expansion and omnimodal alignment. We evaluate Dynin-Omni across 19 multimodal benchmarks spanning language reasoning, image generation and editing, video understanding, and speech recognition and synthesis. Dynin-Omni achieves 87.6 on GSM8K, 1733.6 on MME-P, 61.4 on VideoMME, 0.87 on GenEval, and 2.1 WER on LibriSpeech test-clean, consistently outperforming existing open-source unified models while remaining competitive with strong modality-specific expert systems. These results demonstrate the potential of masked diffusion as a unified paradigm for any-to-any modeling, providing a flexible foundation for real-time omnimodal systems, unified cross-modal retrieval and generation, and embodied multimodal agents.
Abstract (translated)
我们推出Dynin-Omni，这是首个基于掩码扩散的全模态基础模型，能够在单一架构内统一文本、图像和语音的理解与生成，同时涵盖视频理解。与将异质模态序列化的自回归统一模型，或需要外部模态特定解码器编排的组合式统一模型不同，Dynin-Omni将全模态建模原生地构建为共享离散词元空间上的掩码扩散，支持在双向上下文中的迭代精炼。Dynin-Omni采用基于模型合并的模态扩展与全模态对齐的多阶段训练策略。我们在涵盖语言推理、图像生成与编辑、视频理解、语音识别与合成的19个多模态基准上评估Dynin-Omni。Dynin-Omni在GSM8K上达到87.6，在MME-P上达到1733.6，在VideoMME上达到61.4，在GenEval上达到0.87，在LibriSpeech test-clean上达到2.1 WER，持续优于现有开源统一模型，同时与强大的模态特定专家系统保持竞争力。这些结果证明了掩码扩散作为任意到任意建模统一范式的潜力，为实时全模态系统、统一跨模态检索与生成以及具身多模态智能体提供了灵活的基础。
URL
https://arxiv.org/abs/2604.00007
PDF
-
Bootstrapping Audiovisual Speech Recognition in Zero-AV-Resource Scenarios with Synthetic Visual Data
2026-03-09 11:22:24Pol Buitrago, Pol G\`alvez, Oriol Pareras, Javier HernandoAbstract
Audiovisual speech recognition (AVSR) combines acoustic and visual cues to improve transcription robustness under challenging conditions but remains out of reach for most under-resourced languages due to the lack of labeled video corpora for training. We propose a zero-AV-resource AVSR framework that relies on synthetic visual streams generated by lip-syncing static facial images with real audio. We first evaluate synthetic visual augmentation on Spanish benchmarks, then apply it to Catalan, a language with no annotated audiovisual corpora. We synthesize over 700 hours of talking-head video and fine-tune a pre-trained AV-HuBERT model. On a manually annotated Catalan benchmark, our model achieves near state-of-the-art performance with much fewer parameters and training data, outperforms an identically trained audio-only baseline, and preserves multimodal advantages in noise. Scalable synthetic video thus offers a viable substitute for real recordings in zero-AV-resource AVSR.
Abstract (translated)
音频-视频语音识别（AVSR）结合了声学和视觉线索，以在挑战性条件下提高转录的鲁棒性。然而，由于缺乏用于训练的标注视频语料库，大多数资源匮乏的语言仍然无法使用这种技术。我们提出了一种零音频-视频资源的AVSR框架，该框架依靠通过将真实的语音与静态面部图像进行唇同步生成合成视觉流来实现。 首先，我们在西班牙语基准数据集上评估了合成视觉增强的效果；然后将其应用于加泰罗尼亚语——一种没有标注过的音频-视频语料库的语言。我们合成了超过700小时的说话头像视频，并对预训练的AV-HuBERT模型进行了微调。在手动注释的加泰罗尼亚语基准数据集上，我们的模型实现了接近最先进水平的表现，使用了更少的参数和训练数据，在噪声环境下的多模态优势也得到了保持，并且优于相同的音频独立法基线模型。 因此，可扩展的合成视频为零音频-视频资源条件下的AVSR提供了一种可行的替代方案。
URL
https://arxiv.org/abs/2603.08249
PDF
-
Ramsa: A Large Sociolinguistically Rich Emirati Arabic Speech Corpus for ASR and TTS
2026-03-09 09:04:41Rania Al-SabbaghAbstract
Ramsa is a developing 41-hour speech corpus of Emirati Arabic designed to support sociolinguistic research and low-resource language technologies. It contains recordings from structured interviews with native speakers and episodes from national television shows. The corpus features 157 speakers (59 female, 98 male), spans subdialects such as Urban, Bedouin, and Mountain/Shihhi, and covers topics such as cultural heritage, agriculture and sustainability, daily life, professional trajectories, and architecture. It consists of 91 monologic and 79 dialogic recordings, varying in length and recording conditions. A 10\% subset was used to evaluate commercial and open-source models for automatic speech recognition (ASR) and text-to-speech (TTS) in a zero-shot setting to establish initial baselines. Whisper-large-v3-turbo achieved the best ASR performance, with average word and character error rates of 0.268 and 0.144, respectively. MMS-TTS-Ara reported the best mean word and character rates of 0.285 and 0.081, respectively, for TTS. These baselines are competitive but leave substantial room for improvement. The paper highlights the challenges encountered and provides directions for future work.
Abstract (translated)
Ramsa 是一个正在开发中的41小时的阿联酋阿拉伯语语音语料库，旨在支持社会语言学研究和低资源语言技术的发展。该语料库包含与本地母语者的结构化访谈录音以及国家电视台节目的片段。它包括了来自157位说话者（其中女性59人，男性98人）的录音，涵盖了都市、贝都因人和山区/希赫方言等子方言，并涉及文化传统、农业及可持续性、日常生活、职业路径和建筑等话题。该语料库包含91段独白录音和79段对话录音，时长和录音条件各不相同。 为了在零样本设置下评估商业和开源模型的自动语音识别（ASR）和文本到语音（TTS）性能，并建立初步基准线，使用了语料库中10%的数据集进行了测试。Whisper-large-v3-turbo 在 ASR 方面表现最佳，其平均词错误率（WER）为0.268，字符错误率（CER）为0.144；而 MMS-TTS-Ara 报告了 TTS 的最佳均值词和字符速率，分别为 0.285 和 0.081。这些基准虽然具备竞争力，但仍存在很大的改进空间。论文还指出了所遇到的挑战，并为未来的工作提供了方向。
URL
https://arxiv.org/abs/2603.08125
PDF
-
Nw=ach=a Mun=a: A Devanagari Speech Corpus and Proximal Transfer Benchmark for Nepal Bhasha ASR
2026-03-08 09:35:05Rishikesh Kumar Sharma, Safal Narshing Shrestha, Jenny Poudel, Rupak Tiwari, Arju Shrestha, Rupak Raj Ghimire, Bal Krishna BalAbstract
Nepal Bhasha (Newari), an endangered language of the Kathmandu Valley, remains digitally marginalized due to the severe scarcity of annotated speech resources. In this work, we introduce NwÄchÄ MunÄ, a newly curated 5.39-hour manually transcribed Devanagari speech corpus for Nepal Bhasha, and establish the first benchmark using script-preserving acoustic modeling. We investigate whether proximal cross-lingual transfer from a geographically and linguistically adjacent language (Nepali) can rival large-scale multilingual pretraining in an ultra-low-resource Automatic Speech Recognition (ASR) setting. Fine-tuning a Nepali Conformer model reduces the Character Error Rate (CER) from a 52.54% zero-shot baseline to 17.59% with data augmentation, effectively matching the performance of the multilingual Whisper-Small model despite utilizing significantly fewer parameters. Our findings demonstrate that proximal transfer within South Asian language clusters serves as a computationally efficient alternative to massive multilingual models. We openly release the dataset and benchmarks to digitally enable the Newari community and foster further research in Nepal Bhasha.
Abstract (translated)
尼泊尔语（涅瓦利语，Newari）是加德满都谷地的一种濒危语言，由于标注语音资源极度稀缺，其在数字领域一直处于边缘地位。在这项工作中，我们引入了Nwāchā Muna，这是一个新整理的5.39小时的手动转写德瓦那格里语语音数据集，用于尼泊尔语，并建立了第一个使用保持书写形式声学建模基准。本研究探讨了从地理位置和语言上相邻的语言（尼泊尔语）进行邻近跨语言转移，是否能在极度低资源的自动语音识别(ASR)设置中与大规模多语言预训练相匹敌。通过对尼泊尔语Conformer模型进行微调，在使用数据增强的情况下，字符错误率(CER)从零样本基线52.54%降低到17.59%，尽管使用的参数显著较少，但其性能几乎与多语言Whisper-Small模型相当。我们的研究结果表明，在南亚语系中进行邻近转移是大规模多语言模型的一种计算效率更高的替代方案。我们公开发布了该数据集和基准测试，以在数字领域为涅瓦利社区赋能，并促进尼泊尔语进一步的研究。
URL
https://arxiv.org/abs/2603.07554
PDF
-
Seeing the Context: Rich Visual Context-Aware Speech Recognition via Multimodal Reasoning
2026-03-07 15:45:54Wenjie Tian, Mingchen Shao, Bingshen Mu, Xuelong Geng, Chengyou Wang, Yujie Liao, Zhixian Zhao, Ziyu Zhang, Jingbin Hu, Mengqi Wei, Lei XieAbstract
Audio-visual speech recognition (AVSR) is an extension of ASR that incorporates visual signals. Current AVSR approaches primarily focus on lip motion, largely overlooking rich context present in the video such as speaking scene and on-screen text. To tackle such CAVSR (AVSR including rich visual Context), we propose VASR designed to "see" and reason the visual context to improve speech recognition. Specifically, we construct an Audio-Visual Chain-of-Thought (AV-CoT) that explicitly enforces intermediate cross-modal grounding between acoustic signals and visual evidence. This evidence-driven reasoning mitigates the "single-modality dominance" problem, where models either over-rely on visual context or fail to utilize it. Besides, to address the data scarcity, we construct and release a corresponding data pipeline and test set. Experiments show that AV-CoT effectively mitigates the single-modality dominance, achieving state-of-the-art performance in CAVSR. The project is open-sourced.
Abstract (translated)
音频-视觉语音识别（AVSR）是自动语音识别（ASR）的扩展，它结合了视觉信号。目前的AVSR方法主要关注唇部运动，而忽略了视频中存在的丰富背景信息，如说话场景和屏幕上的文本。为了应对包含丰富视觉上下文（CAVSR）的音频-视觉语音识别挑战，我们提出了一种名为VASR的方法，该方法旨在“看到”并推理视觉上下文以提高语音识别性能。具体来说，我们构建了一个音频-视觉链式思考（AV-CoT），它明确地强制执行声学信号与视觉证据之间的中间跨模态基础工作。这种基于证据的推理减轻了“单一模态主导”的问题，即模型要么过度依赖视觉上下文，要么未能有效利用它。 此外，为了解决数据稀缺的问题，我们构建并发布了相应的数据管道和测试集。实验表明，AV-CoT有效地缓解了单一模态主导的问题，在CAVSR方面取得了最先进的性能表现。该项目已经开源。
URL
https://arxiv.org/abs/2603.07263
PDF
-
The Talking Robot: Distortion-Robust Acoustic Models for Robot-Robot Communication
2026-03-07 07:06:41Hanlong Li, Karishma Kamalahasan, Jiahui Li, Kazuhiro Nakadai, Shreyas KousikAbstract
We present Artoo, a learned acoustic communication system for robots that replaces hand-designed signal processing with end-to-end co-trained neural networks. Our system pairs a lightweight text-to-speech (TTS) transmitter (1.18M parameters) with a conformer-based automatic speech recognition (ASR) receiver (938K parameters), jointly optimized through a differentiable channel. Unlike human speech, robot-to-robot communication is paralinguistics-free: the system need not preserve timbre, prosody, or naturalness, only maximize decoding accuracy under channel distortion. Through a three-phase co-training curriculum, the TTS transmitter learns to produce distortion-robust acoustic encodings that surpass the baseline under noise, achieving 8.3% CER at 0 dB SNR. The entire system requires only 2.1M parameters (8.4 MB) and runs in under 13 ms end-to-end on a CPU, making it suitable for deployment on resource-constrained robotic platforms.
Abstract (translated)
我们介绍了Artoo，这是一个学习型声学通信系统，专为机器人设计。该系统用端到端联合训练的神经网络替代了手动设计的信号处理过程。我们的系统将轻量级的文字转语音（TTS）发射器（1.18M参数）与基于卷积变压器（Conformer）的自动语音识别（ASR）接收器（938K参数）配对，通过可微分信道进行联合优化。与人类语言不同，机器人之间的通信无需考虑副语言信息：系统不需要保留音色、语调或自然度，只需在信道失真的情况下最大化解码准确性。 通过一个三阶段的协同训练课程，TTS发射器学会了生成抗干扰能力强的声音编码，在噪声环境下超过了基线表现，实现了0 dB信噪比下的CER（错误率）8.3%。整个系统仅需2.1M参数（8.4MB），并且在CPU上运行时端到端延迟不到13毫秒，非常适合部署于计算资源受限的机器人平台。
URL
https://arxiv.org/abs/2603.07072
PDF
-
Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning
2026-03-06 17:37:06Yuchen Zhang, Haralambos Mouratidis, Ravi ShekharAbstract
Automatic speech recognition (ASR) has benefited from advances in pretrained speech and language models, yet most systems remain constrained to monolingual settings and short, isolated utterances. While recent efforts in context-aware ASR show promise, two key challenges persist: limited multilingual support and the absence of principled alignment between speech and contextual representations. In this paper, we introduce a context-aware multilingual ASR framework that supports diverse languages and accents while preserving the modularity of pretrained models. Our approach combines a frozen speech encoder and a decoder-only language model via a lightweight projection module, allowing structured context prompts, including dialogue history and biasing words, to guide transcription. To improve interaction between speech and context, we employ a contrastive learning objective that aligns their representations in a shared embedding space. Evaluations on over 1,500 hours of real-world conversational speech across 11 languages and 5 English dialects show that contextual input consistently improves recognition quality. Contrastive alignment provides additional gains when applied to different context types, with an overall performance gain of over 5%. These results highlight the importance of both contextual modeling and cross-modal alignment in multilingual ASR.
Abstract (translated)
自动语音识别（ASR）技术受益于预训练的语音和语言模型的进步，但大多数系统仍局限于单语环境和简短孤立的话语。虽然最近在上下文感知ASR方面的努力显示出前景，但仍存在两个关键挑战：对多语言支持有限以及缺乏将语音与上下文表示进行原则性对齐的方法。本文介绍了一种新的上下文感知的多语言ASR框架，该框架支持多种语言和口音的同时保留了预训练模型的模块化特性。我们的方法结合了一个冻结的语音编码器和一个解码器主导的语言模型通过轻量级投影模块连接起来，这使得结构化的上下文提示（包括对话历史记录和倾向词）能够引导转录过程。为了改善语音与上下文之间的交互，我们采用了一种对比学习目标，该目标在共享嵌入空间中对齐它们的表示形式。 通过对跨越11种语言及5种英语方言超过1,500小时的真实会话语音进行评估显示，上下文输入一致提高了识别质量。当应用于不同类型的情况下，对比对齐提供了额外的优势，并且整体性能提高了超过5%。这些结果强调了多语言ASR中上下文建模和跨模式对齐的重要性。
URL
https://arxiv.org/abs/2603.06505
PDF
-
Continual Adaptation for Pacific Indigenous Speech Recognition
2026-03-06 14:13:37Yang Xiao, Aso Mahmudi, Nick Thieberger, Eliathamby Ambikairajah, Eun-Jung Holden, Ting DangAbstract
Speech foundation models struggle with low-resource Pacific Indigenous languages because of severe data scarcity. Furthermore, full fine-tuning risks catastrophic forgetting. To address this gap, we present an empirical study adapting models to real-world Pacific datasets. We investigate how data volume and linguistic features affect adaptation success. Specifically, we evaluate strategies including Full Fine-Tuning and Low-Rank Adaptation (LoRA). Additionally, we analyze a continual learning framework for sequentially acquiring multiple languages. We demonstrate that adapting to these distant languages causes severe internal representational drift. Consequently, these models face a strict plasticity and stability dilemma. While LoRA adapts well initially, it suffers from catastrophic forgetting during sequential learning. Ultimately, this study highlights the urgent need for robust adaptation strategies tailored to underrepresented languages.
Abstract (translated)
语音基础模型在处理低资源的太平洋土著语时面临挑战，主要是由于数据稀缺。此外，进行全面微调会带来灾难性遗忘的风险。为了解决这一问题，我们进行了一项实证研究，探讨如何将这些模型适应于现实世界的太平洋语言数据集。我们的研究考察了数据量和语言特征对适应效果的影响，并评估了几种策略的有效性，包括全面微调（Full Fine-Tuning）和低秩自适应（Low-Rank Adaptation, LoRA）。此外，我们还分析了一个连续学习框架，该框架能够顺序地获取多种语言。我们的研究发现，在这些远距离的语言上进行适应会导致严重的内部表示漂移问题。因此，这些模型面临着严格的可塑性和稳定性困境。 虽然LoRA在初始阶段表现良好，但在顺序学习过程中会遭遇灾难性遗忘的问题。最终，这项研究表明，急需为那些代表性不足的土著语言制定稳健的自适应策略。
URL
https://arxiv.org/abs/2603.06310
PDF
-
Whisper-CD: Accurate Long-Form Speech Recognition using Multi-Negative Contrastive Decoding
2026-03-06 12:04:12Hoseong Ahn, Jeongyun Chae, Yoonji Park, Kyuhong ShimAbstract
Long-form speech recognition with large encoder-decoder models such as Whisper often exhibit hallucinations, repetition loops, and content omissions. These errors can accumulate and be further amplified when the previous segment's transcription is used as decoding context. We propose Whisper-CD, a training-free contrastive decoding framework that contrasts clean-audio logits against negative logits computed from three acoustically motivated perturbations: Gaussian noise injection, silence signal, and audio temporal shift. We aggregate these negatives via the log-sum-exp operator, building a unified multi-negative objective for token-by-token decoding. Across five English long-form benchmarks, Whisper-CD reduces WER by up to 24.3pp on CORAAL and shows 48% faster token generation throughput than beam search. Because Whisper-CD operates purely at inference time, it can be applied as a drop-in replacement to already-deployed Whisper systems without retraining.
Abstract (translated)
长语音识别中，使用如Whisper这样的大型编码器解码器模型时常会出现幻觉、重复循环和内容遗漏等问题。这些问题在前一段转录被用作解码上下文时会积累并进一步放大。我们提出了一种无需训练的对比解码框架——Whisper-CD，该框架通过将清洁音频的概率向量与三种声学动机的扰动计算出的负样本概率向量进行对比来减少错误：高斯噪声注入、静音信号以及音频时间偏移。这些负样本通过对数和指数运算聚合在一起，形成了一种统一的多负样本目标函数，用于逐令牌解码过程。 在五个英文长语音基准数据集上，Whisper-CD将WER（单词错误率）降低了最多24.3个百分点（CORAAL数据集中），并且比束搜索方法快48%。由于Whisper-CD仅在推理阶段工作，因此可以在不重新训练的情况下直接替换已部署的Whisper系统。
URL
https://arxiv.org/abs/2603.06193
PDF
-
Which Data Matter? Embedding-Based Data Selection for Speech Recognition
2026-03-06 02:07:08Zakaria Aldeneh, Skyler Seto, Maureen de Seyssel, Jie Chi, Zijin Gu, Takuya Higuchi, Jee-weon Jung, Shinji Watanabe, David Grangier, Barry-John Theobald, Tatiana LikhomanenkoAbstract
Modern ASR systems are typically trained on large-scale pseudo-labeled, in-the-wild data spanning multiple domains. While such heterogeneous data benefit generalist models designed for broad deployment, they pose challenges for specialist models targeting specific domains: specialist models lack the capacity to learn from all available data, and one must pay closer attention to addressing the mismatch between training and test conditions. In this work, we study targeted data selection as a strategy to address these challenges, selecting relevant subsets from 100k hours of in-the-wild training data to optimize performance on target domains. We represent speech samples using embeddings that capture complementary characteristic--speaker attributes, phonetic content, and semantic meaning--and analyze how relevance and diversity along these axes when performing data selection affect downstream ASR performance. Our experiments with CTC-based Conformer models show that training on a strategically selected 5% subset can exceed the performance of models trained on the full dataset by up to 36.8% relative WER reduction on target domains.
Abstract (translated)
现代的自动语音识别（ASR）系统通常是在大规模伪标注的真实场景数据上进行训练，这些数据涵盖了多个领域。虽然这种异质性数据对旨在广泛部署的一般模型有利，但对于针对特定领域的专业模型来说却带来了挑战：专业模型缺乏从所有可用数据中学习的能力，并且必须更加注意解决训练和测试条件之间的不匹配问题。在这项工作中，我们研究了有目标的数据选择策略来应对这些挑战，即从10万小时的真实场景训练数据中挑选出相关的子集以优化特定领域的性能。我们将语音样本表示为捕获互补特征的嵌入向量——包括说话人属性、音素内容和语义意义，并分析在进行数据选择时沿这些轴的相关性和多样性如何影响下游ASR性能。我们的实验表明，基于CTC的Conformer模型在经过精心挑选的5%子集上训练时，在目标领域中的相对错误词率（WER）可以比在完整数据集上训练的模型降低高达36.8%，这证明了有目标的数据选择策略的有效性。
URL
https://arxiv.org/abs/2603.05819
PDF
-
Berta: an open-source, modular tool for AI-enabled clinical documentation
2026-03-05 19:17:21Samridhi Vaid, Mike Weldon, Jesse Dunn, Sacha Davis, Kevin Lonergan, Henry Li, Jeffrey Franc, Mohamed Abdalla, Daniel C. Baumgart, Jake Hayward, J Ross MitchellAbstract
Commercial AI scribes cost \$99-600 per physician per month, operate as opaque systems, and do not return data to institutional infrastructure, limiting organizational control over data governance, quality improvement, and clinical workflows. We developed Berta, an open-source modular scribe platform for AI-enabled clinical documentation, and deployed a customized implementation within Alberta Health Services (AHS) integrated with their existing Snowflake AI Data Cloud infrastructure. The system combines automatic speech recognition with large language models while retaining all clinical data within the secure AHS environment. During eight months (November 2024 to July 2025), 198 emergency physicians used the system in 105 urban and rural facilities, generating 22148 clinical sessions and more than 2800 hours of audio. The use grew from 680 to 5530 monthly sessions. Operating costs averaged less than \$30 per physician per month, a 70-95% reduction compared to commercial alternatives. AHS has since approved expansion to 850 physicians. This is the first provincial-scale deployment of an AI scribe integrated with existing health system infrastructure. By releasing Berta as open source, we provide a reproducible, cost-effective alternative that health systems can adapt to their own secure environments, supporting data sovereignty and informed evaluation of AI documentation technology.
Abstract (translated)
商业AI文书助手每名医生每月费用为99至600美元，其系统运作不透明，且不向机构基础设施返还数据，从而限制了组织对数据治理、质量改进和临床工作流的控制。我们开发了Berta——一个用于AI辅助临床文档记录的开源模块化文书平台，并在阿尔伯塔省卫生服务局（AHS）内部署了定制化实施方案，该方案与其现有的Snowflake AI数据云基础设施相集成。该系统将自动语音识别与大语言模型相结合，同时将所有临床数据保留在安全的AHS环境中。在八个月期间（2024年11月至2025年7月），198名急诊科医生在105个城市和农村医疗机构中使用该系统，产生了22148次临床会话和超过2800小时的音频。使用量从每月680次会话增长至5530次。平均每名医生每月运营成本低于30美元，相比商业替代方案降低了70%至95%。AHS已批准将系统扩展至850名医生。这是首个与现有卫生系统基础设施集成的省级规模AI文书助手部署。通过将Berta作为开源项目发布，我们提供了一个可复现、成本效益高的替代方案，卫生系统可将其适配至自身安全环境，从而支持数据主权以及对AI文档技术的知情评估。
URL
https://arxiv.org/abs/2603.23513
PDF
-
PersianPunc: A Large-Scale Dataset and BERT-Based Approach for Persian Punctuation Restoration
2026-03-05 15:55:55Mohammad Javad Ranjbar Kalahroodi, Heshaam Faili, Azadeh ShakeryAbstract
Punctuation restoration is essential for improving the readability and downstream utility of automatic speech recognition (ASR) outputs, yet remains underexplored for Persian despite its importance. We introduce PersianPunc, a large-scale, high-quality dataset of 17 million samples for Persian punctuation restoration, constructed through systematic aggregation and filtering of existing textual resources. We formulate punctuation restoration as a token-level sequence labeling task and fine-tune ParsBERT to achieve strong performance. Through comparative evaluation, we demonstrate that while large language models can perform punctuation restoration, they suffer from critical limitations: over-correction tendencies that introduce undesired edits beyond punctuation insertion (particularly problematic for speech-to-text pipelines) and substantially higher computational requirements. Our lightweight BERT-based approach achieves a macro-averaged F1 score of 91.33% on our test set while maintaining efficiency suitable for real-time applications. We make our dataset (this https URL) and model (this https URL) publicly available to facilitate future research in Persian NLP and provide a scalable framework applicable to other morphologically rich, low-resource languages.
Abstract (translated)
标点符号恢复对于提高自动语音识别（ASR）输出的可读性和下游应用的实用性至关重要，然而，在波斯语中这一领域尚未得到充分探索。我们推出了PersianPunc，这是一个大规模、高质量的数据集，包含1700万个样本，旨在解决波斯语文本标点符号恢复问题，通过系统性地聚合和筛选现有文本资源构建而成。我们将标点符号恢复定义为一个标记级序列标注任务，并针对此任务微调了ParsBERT模型以实现强大性能。通过比较评估我们发现，虽然大型语言模型能够执行标点符号恢复，但它们存在关键局限：过度校正倾向导致引入除插入标点之外的不必要修改（特别是在语音转文本管道中尤为严重），并且需要更高的计算资源。我们的基于轻量级BERT的方法在测试集上实现了91.33%的宏平均F1分数，并且保持了适用于实时应用的效率。 我们公开发布了数据集（[此链接](https://this https URL)）和模型（[此链接](https://this https URL)），以促进未来波斯语自然语言处理领域的研究，同时也为其他形态丰富、资源有限的语言提供了一种可扩展框架。
URL
https://arxiv.org/abs/2603.05314
PDF
-
Visual-Informed Speech Enhancement Using Attention-Based Beamforming
2026-03-05 15:19:41Chihyun Liu, Jiaxuan Fan, Mingtung Sun, Michael Anthony, Mingsian R. Bai, Yu TsaoAbstract
Recent studies have demonstrated that incorporating auxiliary information, such as speaker voiceprint or visual cues, can substantially improve Speech Enhancement (SE) performance. However, single-channel methods often yield suboptimal results in low signal-to-noise ratio (SNR) conditions, when there is high reverberation, or in complex scenarios involving dynamic speakers, overlapping speech, or non-stationary noise. To address these issues, we propose a novel Visual-Informed Neural Beamforming Network (VI-NBFNet), which integrates microphone array signal processing and deep neural networks (DNNs) using multimodal input features. The proposed network leverages a pretrained visual speech recognition model to extract lip movements as input features, which serve for voice activity detection (VAD) and target speaker identification. The system is intended to handle both static and moving speakers by introducing a supervised end-to-end beamforming framework equipped with an attention mechanism. The experimental results demonstrated that the proposed audiovisual system has achieved better SE performance and robustness for both stationary and dynamic speaker scenarios, compared to several baseline methods.
Abstract (translated)
最近的研究表明，整合辅助信息（如说话人的声纹或视觉线索）可以显著提高语音增强(SE)的性能。然而，在低信噪比(SNR)条件、存在高混响情况或是涉及动态发言者、重叠讲话以及非平稳噪声等复杂场景下，单通道方法通常会导致次优结果。为解决这些问题，我们提出了一种新颖的视觉引导神经波束成形网络(VI-NBFNet)，该网络将麦克风阵列信号处理与深度神经网络(DNN)相结合，并利用多模态输入特征进行操作。 所提出的网络采用了预训练的视觉语音识别模型来提取唇部运动作为输入特征，这些特征用于语音活动检测(VAD)和目标发言者识别。通过引入一种配备注意力机制的监督端到端波束成形框架，该系统旨在处理静态和移动发言者的场景。实验结果表明，在与几种基线方法相比时，所提出的视听系统在静止和动态发言者的场景下均实现了更好的语音增强性能和鲁棒性。
URL
https://arxiv.org/abs/2603.05270
PDF
-
Boosting ASR Robustness via Test-Time Reinforcement Learning with Audio-Text Semantic Rewards
2026-03-05 14:43:15Linghan Fang, Tianxin Xie, Li LiuAbstract
Recently, Automatic Speech Recognition (ASR) systems (e.g., Whisper) have achieved remarkable accuracy improvements but remain highly sensitive to real-world unseen data (data with large distribution shifts), including noisy environments and diverse accents. To address this issue, test-time adaptation (TTA) has shown great potential in improving the model adaptability at inference time without ground-truth labels, and existing TTA methods often rely on pseudo-labeling or entropy minimization. However, by treating model confidence as a learning signal, these methods may reinforce high-confidence errors, leading to confirmation bias that undermines adaptation. To overcome these limitations, we present ASR-TRA, a novel Test-time Reinforcement Adaptation framework inspired by causal intervention. More precisely, our method introduces a learnable decoder prompt and utilizes temperature-controlled stochastic decoding to generate diverse transcription candidates. These are scored by a reward model that measures audio-text semantic alignment, and the resulting feedback is used to update both model and prompt parameters via reinforcement learning. Comprehensive experiments on LibriSpeech with synthetic noise and L2 Arctic accented English datasets demonstrate that our method achieves higher accuracy while maintaining lower latency than existing TTA baselines. Ablation studies further confirm the effectiveness of combining audio and language-based rewards, highlighting our method's enhanced stability and interpretability. Overall, our approach provides a practical and robust solution for deploying ASR systems in challenging real-world conditions.
Abstract (translated)
最近，自动语音识别（ASR）系统（例如Whisper）在准确性方面取得了显著的改进，但仍然对现实世界中的未见数据（具有较大分布变化的数据），如噪音环境和多样化的口音非常敏感。为了解决这个问题，在推断时间进行测试适应性调整（TTA）展示了极大的潜力，可以在没有真实标签的情况下提高模型的适应性。现有的TTA方法通常依赖于伪标记或熵最小化。然而，通过将模型置信度视为学习信号，这些方法可能会强化高置信度错误，导致确认偏差从而削弱了适应能力。 为了克服这些限制，我们提出了ASR-TRA，这是一种受因果干预启发的新型测试时间增强适应框架。具体而言，我们的方法引入了一个可学习的解码器提示，并利用温度控制的随机解码来生成多样化的转录候选方案。通过一个奖励模型对音频和文本语义一致性进行评分，这些候选方案被评估并反馈用于通过强化学习更新模型参数和提示参数。在LibriSpeech合成噪音数据集以及L2 Arctic口音英语数据集中进行了全面实验，结果表明我们的方法相较于现有的TTA基线，在保持较低延迟的情况下实现了更高的准确性。 消融研究进一步证实了结合音频和语言奖励的有效性，突显出我们方法增强了稳定性和可解释性。总体而言，我们的方法为在具有挑战性的现实条件下部署ASR系统提供了实用且稳健的解决方案。
URL
https://arxiv.org/abs/2603.05231
PDF
-
PCOV-KWS: Multi-task Learning for Personalized Customizable Open Vocabulary Keyword Spotting
2026-03-05 13:57:52Jianan Pan, Kejie HuangAbstract
As advancements in technologies like Internet of Things (IoT), Automatic Speech Recognition (ASR), Speaker Verification (SV), and Text-to-Speech (TTS) lead to increased usage of intelligent voice assistants, the demand for privacy and personalization has escalated. In this paper, we introduce a multi-task learning framework for personalized, customizable open-vocabulary Keyword Spotting (PCOV-KWS). This framework employs a lightweight network to simultaneously perform Keyword Spotting (KWS) and SV to address personalized KWS requirements. We have integrated a training criterion distinct from softmax-based loss, transforming multi-class classification into multiple binary classifications, which eliminates inter-category competition, while an optimization strategy for multi-task loss weighting is employed during training. We evaluated our PCOV-KWS system in multiple datasets, demonstrating that it outperforms the baselines in evaluation results, while also requiring fewer parameters and lower computational resources.
Abstract (translated)
URL
https://arxiv.org/abs/2603.18023
PDF
-
Federated Heterogeneous Language Model Optimization for Hybrid Automatic Speech Recognition
2026-03-05 08:42:17Mengze Hong, Yi Gu, Di Jiang, Hanlin Gu, Chen Jason Zhang, Lu Wang, Zhiyang SuAbstract
Training automatic speech recognition (ASR) models increasingly relies on decentralized federated learning to ensure data privacy and accessibility, producing multiple local models that require effective merging. In hybrid ASR systems, while acoustic models can be merged using established methods, the language model (LM) for rescoring the N-best speech recognition list faces challenges due to the heterogeneity of non-neural n-gram models and neural network models. This paper proposes a heterogeneous LM optimization task and introduces a match-and-merge paradigm with two algorithms: the Genetic Match-and-Merge Algorithm (GMMA), using genetic operations to evolve and pair LMs, and the Reinforced Match-and-Merge Algorithm (RMMA), leveraging reinforcement learning for efficient convergence. Experiments on seven OpenSLR datasets show RMMA achieves the lowest average Character Error Rate and better generalization than baselines, converging up to seven times faster than GMMA, highlighting the paradigm's potential for scalable, privacy-preserving ASR systems.
Abstract (translated)
训练自动语音识别（ASR）模型越来越依赖于分散式的联邦学习，以确保数据隐私和可访问性，并生成需要有效合并的多个本地模型。在混合ASR系统中，虽然可以使用已建立的方法合并声学模型，但用于重评分N-best语音识别列表的语言模型（LM）由于非神经n-gram模型与神经网络模型之间的异质性而面临挑战。 本文提出了一项针对异构LM优化的任务，并引入了匹配和合并范式，包含两种算法：使用遗传操作进化并配对语言模型的遗传匹配与合并算法（GMMA），以及利用强化学习实现高效收敛的强化匹配与合并算法（RMMA）。在七个OpenSLR数据集上的实验表明，RMMA达到了最低的平均字符错误率，并且比基线方法有更好的泛化能力，其收敛速度是GMMA的七倍快，这突显了该范式在构建可扩展和保护隐私的ASR系统中的潜力。
URL
https://arxiv.org/abs/2603.04945
PDF
-
WhisperAlign: Word-Boundary-Aware ASR and WhisperX-Anchored Pyannote Diarization for Long-Form Bengali Speech
2026-03-05 04:54:11Aurchi Chowdhury, Rubaiyat -E-Zaman, Sk. Ashrafuzzaman NafeesAbstract
This paper presents our solution for the DL Sprint 4.0, addressing the dual challenges of Bengali Long-Form Speech Recognition (Task 1) and Speaker Diarization (Task 2). Processing long-form, multi-speaker Bengali audio introduces significant hurdles in voice activity detection, overlapping speech, and context preservation. To solve the long-form transcription challenge, we implemented a robust audio chunking strategy utilizing whisper-timestamped, allowing us to feed precise, context-aware segments into our fine-tuned acoustic model for high-accuracy transcription. For the diarization task, we developed an integrated pipeline leveraging this http URL and WhisperX. A key contribution of our approach is the domain-specific fine-tuning of the Pyannote segmentation model on the competition dataset. This adaptation allowed the model to better capture the nuances of Bengali conversational dynamics and accurately resolve complex, overlapping speaker boundaries. Our methodology demonstrates that applying intelligent timestamped chunking to ASR and targeted segmentation fine-tuning to diarization significantly drives down Word Error Rate (WER) and Diarization Error Rate (DER), in low-resource settings.
Abstract (translated)
本文介绍了我们针对DL Sprint 4.0竞赛提出的解决方案，该方案旨在解决孟加拉语长语音识别（任务1）和说话人角色划分（任务2）的双重挑战。处理多说话人的孟加拉语文本带来了诸如语音活动检测、重叠言语以及上下文保持等方面的显著难题。 为了解决长文本转录的问题，我们实施了一种稳健的声音片段策略，利用了带有时间戳的whisper音频切分技术，使我们可以将精确且包含上下文信息的片段输入到经过微调的声学模型中以实现高精度转录。对于说话人角色划分的任务，我们开发了一个集成管道，结合使用上述方法和WhisperX工具。 我们的方法的一个关键贡献是，在竞赛数据集上对Pyannote分段模型进行了领域特定的微调。这种调整使模型能够更好地捕捉孟加拉语对话动态中的细微差别，并准确解析复杂的重叠说话人边界问题。我们的方法表明，将智能时间戳切片技术应用于自动语音识别（ASR）以及针对角色划分进行目标细分微调，在资源有限的情况下可以显著降低单词错误率（WER）和角色划分误差率（DER）。
URL
https://arxiv.org/abs/2603.04809
PDF
-
When Denoising Hinders: Revisiting Zero-Shot ASR with SAM-Audio and Whisper
2026-03-05 01:20:11Akif Islam, Raufun Nahar, Md. Ekramul HamidAbstract
Recent advances in automatic speech recognition (ASR) and speech enhancement have led to a widespread assumption that improving perceptual audio quality should directly benefit recognition accuracy. In this work, we rigorously examine whether this assumption holds for modern zero-shot ASR systems. We present a systematic empirical study on the impact of Segment Anything Model Audio by Meta AI, a recent foundation-scale speech enhancement model proposed by Meta, when used as a preprocessing step for zero-shot transcription with Whisper. Experiments are conducted across multiple Whisper model variants and two linguistically distinct noisy speech datasets: a real-world Bengali YouTube corpus and a publicly available English noisy dataset. Contrary to common intuition, our results show that SAM-Audio preprocessing consistently degrades ASR performance, increasing both Word Error Rate (WER) and Character Error Rate (CER) compared to raw noisy speech, despite substantial improvements in signal-level quality. Objective Peak Signal-to-Noise Ratio analysis on the English dataset confirms that SAM-Audio produces acoustically cleaner signals, yet this improvement fails to translate into recognition gains. Therefore, we conducted a detailed utterance-level analysis to understand this counterintuitive result. We found that the recognition degradation is a systematic issue affecting the majority of the audio, not just isolated outliers, and that the errors worsen as the Whisper model size increases. These findings expose a fundamental mismatch: audio that is perceptually cleaner to human listeners is not necessarily robust for machine recognition. This highlights the risk of blindly applying state-of-the-art denoising as a preprocessing step in zero-shot ASR pipelines.
Abstract (translated)
近期在自动语音识别（ASR）和语音增强领域的进展导致了一种广泛的看法，即提高感知音频质量应直接有助于提升识别准确性。然而，在这项工作中，我们严谨地检验了这种假设是否适用于现代的零样本ASR系统。我们对Meta AI提出的最近一种大规模基础模型——Segment Anything Model Audio (SAM-Audio)进行了一项系统的实证研究，该模型用作使用Whisper进行零样本转录的预处理步骤时的影响。实验在多个Whisper模型变体以及两种语言不同的嘈杂语音数据集上进行：一个是现实世界的孟加拉语YouTube语料库，另一个是公开可用的英语嘈杂数据集。 与常见的直觉相反，我们的结果显示SAM-Audio预处理一致地降低了ASR性能，相比于原始的嘈杂音频，无论是词错误率（WER）还是字符错误率（CER）都有所增加。尽管信号级质量有了显著提升，但这种改进并未转化为识别效果上的增益。对英语数据集进行客观峰值信噪比分析表明，SAM-Audio确实产生了音质更清晰的信号，然而这一改善未能转化成实际的识别收益。 因此，我们进行了详细的句子级别分析以理解这一反直觉的结果。发现识别性能下降是一个影响大部分音频的系统性问题，而不仅仅是孤立的异常值，并且随着Whisper模型规模的增长，错误率也随之增加。这些发现在根本上揭示了这样一种矛盾：对于人类听者来说更为清晰的音频不一定是机器识别时更加稳健的信号。 这项研究强调了在零样本ASR流程中盲目使用最先进的降噪技术作为预处理步骤的风险，并提出了挑战当前假设的重要见解，即优化感知音质并不一定意味着能够直接提升语音识别性能。
URL
https://arxiv.org/abs/2603.04710
PDF
