Speech_Recognition
-
Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild
2026-03-28 05:09:16Berkin Durmus, Chen Cen, Eduardo Pacheco, Arda Okan, Atila OrhonAbstract
The accuracy frontier of speech-to-text systems has plateaued on academic benchmarks.1 In contrast, industrial benchmarks and adoption in high-stakes domains suggest otherwise. We hypothesize that the primary difference between the two is contextual conditioning: Academic benchmarks are dominated by frequently encountered general vocabulary that is relatively easy to recognize compared with rare and context-defined custom vocabulary that has disproportionate impact on the usability of speech transcripts. Despite progress on contextual speech-to-text, there is no standardized benchmark. We introduce Contextual Earnings-22, an open dataset built upon Earnings-22, with realistic custom vocabulary contexts to foster research and reveal latent progress. We set six strong baselines for two dominant approaches: keyword prompting and keyword boosting. Experiments show both reach comparable and significantly improved accuracy when scaled from proof-of-concept to large-scale systems.
Abstract (translated)
URL
https://arxiv.org/abs/2604.07354
PDF
-
JAL-Turn: Joint Acoustic-Linguistic Modeling for Real-Time and Robust Turn-Taking Detection in Full-Duplex Spoken Dialogue Systems
2026-03-27 15:25:38Guangzhao Yang, Yu Pan, Shi Qiu, Ningjie BaiAbstract
Despite recent advances, efficient and robust turn-taking detection remains a significant challenge in industrial-grade Voice AI agent deployments. Many existing systems rely solely on acoustic or semantic cues, leading to suboptimal accuracy and stability, while recent attempts to endow large language models with full-duplex capabilities require costly full-duplex data and incur substantial training and deployment overheads, limiting real-time performance. In this paper, we propose JAL-Turn, a lightweight and efficient speech-only turn-taking framework that adopts a joint acoustic-linguistic modeling paradigm, in which a cross-attention module adaptively integrates pre-trained acoustic representations with linguistic features to support low-latency prediction of hold vs shift states. By sharing a frozen ASR encoder, JAL-Turn enables turn-taking prediction to run fully in parallel with speech recognition, introducing no additional end-to-end latency or computational overhead. In addition, we introduce a scalable data construction pipeline that automatically derives reliable turn-taking labels from large-scale real-world dialogue corpora. Extensive experiments on public multilingual benchmarks and an in-house Japanese customer-service dataset show that JAL-Turn consistently outperforms strong state-of-the-art baselines in detection accuracy while maintaining superior real-time performance.
Abstract (translated)
尽管近期有所进展，但在工业级语音AI代理部署中，高效且鲁棒的轮流对话检测仍是一项重大挑战。许多现有系统仅依赖声学或语义线索，导致准确度与稳定性不足；而近期尝试赋予大语言模型全双工能力的方法，则需要昂贵的全双工数据，并产生巨大的训练与部署开销，从而限制了实时性能。本文提出JAL-Turn，一个轻量高效的纯语音轮流对话框架，采用声学-语言联合建模范式，通过跨注意力模块自适应地融合预训练声学表征与语言特征，以支持低延迟的“保持”与“转换”状态预测。通过共享一个冻结的自动语音识别编码器，JAL-Turn使轮流对话预测能与语音识别完全并行运行，不引入额外的端到端延迟或计算开销。此外，我们引入了一个可扩展的数据构建流程，能够从大规模真实对话语料中自动推导可靠的轮流对话标签。在公开多语言基准数据集和内部日语客服数据集上的大量实验表明，JAL-Turn在检测准确度上持续优于强大的最新基线方法，同时保持卓越的实时性能。
URL
https://arxiv.org/abs/2603.26515
PDF
-
Automatic Speech Recognition for Documenting Endangered Languages: Case Study of Ikema Miyakoan
2026-03-27 10:12:26Chihiro Taguchi, Yukinori Takubo, David ChiangAbstract
Language endangerment poses a major challenge to linguistic diversity worldwide, and technological advances have opened new avenues for documentation and revitalization. Among these, automatic speech recognition (ASR) has shown increasing potential to assist in the transcription of endangered language data. This study focuses on Ikema, a severely endangered Ryukyuan language spoken in Okinawa, Japan, with approximately 1,300 remaining speakers, most of whom are over 60 years old. We present an ongoing effort to develop an ASR system for Ikema based on field recordings. Specifically, we (1) construct a {\totaldatasethours}-hour speech corpus from field recordings, (2) train an ASR model that achieves a character error rate as low as 15\%, and (3) evaluate the impact of ASR assistance on the efficiency of speech transcription. Our results demonstrate that ASR integration can substantially reduce transcription time and cognitive load, offering a practical pathway toward scalable, technology-supported documentation of endangered languages.
Abstract (translated)
URL
https://arxiv.org/abs/2603.26248
PDF
-
Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR
2026-03-27 10:09:30Shashi Kumar, Esa\'u Villatoro-Tello, Sergio Burdisso, Kadri Hacioglu, Thibault Ba\~neras-Roux, Hasindri Watawana, Dairazalia Sanchez-Cortes, Srikanth Madikeri, Petr Motlicek, Andreas StolckeAbstract
Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.
Abstract (translated)
基于标准大语言模型的语音识别系统通常孤立处理话语，限制了其利用对话上下文的能力。本研究探讨了多模态的先前轮次上下文是否能提升基于大语言模型的自动语音识别效果，以及如何高效表征该上下文。我们发现，经过监督式多轮训练后，对话上下文主要有助于识别上下文相关实体。然而，直接使用原始上下文成本高昂，因为先前轮次的音频标记序列会随对话长度快速增长。为此，我们提出抽象压缩方法，用固定数量的学习隐式标记替换先前轮次的音频部分，同时显式保留对应文本转写。在同领域和跨领域测试集上，压缩模型以更小的先前轮次音频占用空间，部分恢复了原始上下文条件设置带来的性能提升。我们还针对压缩配置及其权衡进行了针对性分析。
URL
https://arxiv.org/abs/2603.26246
PDF
-
Goodness-of-pronunciation without phoneme time alignment
2026-03-26 08:12:19Jeremy H. M. Wong, Nancy F. ChenAbstract
In speech evaluation, an Automatic Speech Recognition (ASR) model often computes time boundaries and phoneme posteriors for input features. However, limited data for ASR training hinders expansion of speech evaluation to low-resource languages. Open-source weakly-supervised models are capable of ASR over many languages, but they are frame-asynchronous and not phonemic, hindering feature extraction for speech evaluation. This paper proposes to overcome incompatibilities for feature extraction with weakly-supervised models, easing expansion of speech evaluation to low-resource languages. Phoneme posteriors are computed by mapping ASR hypotheses to a phoneme confusion network. Word instead of phoneme-level speaking rate and duration are used. Phoneme and frame-level features are combined using a cross-attention architecture, obviating phoneme time alignment. This performs comparably with standard frame-synchronous features on English speechocean762 and low-resource Tamil datasets.
Abstract (translated)
在语音评估中，自动语音识别（ASR）模型通常会对输入特征计算时间边界和音素后验概率。然而，ASR训练数据有限阻碍了语音评估向低资源语言的扩展。开源弱监督模型虽能支持多语言ASR，但其帧异步且非音素级的特性阻碍了语音评估的特征提取。本文提出克服弱监督模型在特征提取上的不兼容性，以促进语音评估向低资源语言扩展。通过将ASR假设映射至音素混淆网络来计算音素后验概率，并采用词级而非音素级的语速和时长特征。利用交叉注意力架构融合音素与帧级特征，无需音素时间对齐。该方法在英语speechocean762和低资源泰米尔语数据集上的表现与标准帧同步特征相当。
URL
https://arxiv.org/abs/2603.25150
PDF
-
A Sociolinguistic Analysis of Automatic Speech Recognition Bias in Newcastle English
2026-03-25 17:24:47Dana Serditova, Kevin TangAbstract
Automatic Speech Recognition (ASR) systems are widely used in everyday communication, education, healthcare, and industry, yet their performance remains uneven across speakers, particularly when dialectal variation diverges from the mainstream accents represented in training data. This study investigates ASR bias through a sociolinguistic analysis of Newcastle English, a regional variety of North-East England that has been shown to challenge current speech recognition technologies. Using spontaneous speech from the Diachronic Electronic Corpus of Tyneside English (DECTE), we evaluate the output of a state-of-the-art commercial ASR system and conduct a fine-grained analysis of more than 3,000 transcription errors. Errors are classified by linguistic domain and examined in relation to social variables including gender, age, and socioeconomic status. In addition, an acoustic case study of selected vowel features demonstrates how gradient phonetic variation contributes directly to misrecognition. The results show that phonological variation accounts for the majority of errors, with recurrent failures linked to dialect-specific features like vowel quality and glottalisation, as well as local vocabulary and non-standard grammatical forms. Error rates also vary across social groups, with higher error frequencies observed for men and for speakers at the extremes of the age spectrum. These findings indicate that ASR errors are not random but socially patterned and can be explained from a sociolinguistic perspective. Thus, the study demonstrates the importance of incorporating sociolinguistic expertise into the evaluation and development of speech technologies and argues that more equitable ASR systems require explicit attention to dialectal variation and community-based speech data.
Abstract (translated)
URL
https://arxiv.org/abs/2603.24549
PDF
-
When AI Meets Early Childhood Education: Large Language Models as Assessment Teammates in Chinese Preschools
2026-03-25 15:05:34Xingming Li, Runke Huang, Yanan Bao, Yuye Jin, Yuru Jiao, Qingyong HuAbstract
High-quality teacher-child interaction (TCI) is fundamental to early childhood development, yet traditional expert-based assessment faces a critical scalability challenge. In large systems like China's-serving 36 million children across 250,000+ kindergartens-the cost and time requirements of manual observation make continuous quality monitoring infeasible, relegating assessment to infrequent episodic audits that limit timely intervention and improvement tracking. In this paper, we investigate whether AI can serve as a scalable assessment teammate by extracting structured quality indicators and validating their alignment with human expert judgments. Our contributions include: (1) TEPE-TCI-370h (Tracing Effective Preschool Education), the first large-scale dataset of naturalistic teacher-child interactions in Chinese preschools (370 hours, 105 classrooms) with standardized ECQRS-EC and SSTEW annotations; (2) We develop Interaction2Eval, a specialized LLM-based framework addressing domain-specific challenges-child speech recognition, Mandarin homophone disambiguation, and rubric-based reasoning-achieving up to 88% agreement; (3) Deployment validation across 43 classrooms demonstrating an 18x efficiency gain in the assessment workflow, highlighting its potential for shifting from annual expert audits to monthly AI-assisted monitoring with targeted human oversight. This work not only demonstrates the technical feasibility of scalable, AI-augmented quality assessment but also lays the foundation for a new paradigm in early childhood education-one where continuous, inclusive, AI-assisted evaluation becomes the engine of systemic improvement and equitable growth.
Abstract (translated)
URL
https://arxiv.org/abs/2603.24389
PDF
-
Bridging Biological Hearing and Neuromorphic Computing: End-to-End Time-Domain Audio Signal Processing with Reservoir Computing
2026-03-25 13:17:30Rinku Sebastian, Simon O'Keefe, Martin TrefzerAbstract
Despite the advancements in cutting-edge technologies, audio signal processing continues to pose challenges and lacks the precision of a human speech processing system. To address these challenges, we propose a novel approach to simplify audio signal processing by leveraging time-domain techniques and reservoir computing. Through our research, we have developed a real-time audio signal processing system by simplifying audio signal processing through the utilization of reservoir computers, which are significantly easier to train. Feature extraction is a fundamental step in speech signal processing, with Mel Frequency Cepstral Coefficients (MFCCs) being a dominant choice due to their perceptual relevance to human hearing. However, conventional MFCC extraction relies on computationally intensive time-frequency transformations, limiting efficiency in real-time applications. To address this, we propose a novel approach that leverages reservoir computing to streamline MFCC extraction. By replacing traditional frequency-domain conversions with convolution operations, we eliminate the need for complex transformations while maintaining feature discriminability. We present an end-to-end audio processing framework that integrates this method, demonstrating its potential for efficient and real-time speech analysis. Our results contribute to the advancement of energy-efficient audio processing technologies, enabling seamless deployment in embedded systems and voice-driven applications. This work bridges the gap between biologically inspired feature extraction and modern neuromorphic computing, offering a scalable solution for next-generation speech recognition systems.
Abstract (translated)
尽管尖端技术不断进步，音频信号处理仍面临诸多挑战，且其精度远不及人类语音处理系统。为应对这些挑战，我们提出了一种利用时域技术和储备计算简化音频信号处理的新方法。通过本研究，我们借助更易训练的储备计算机简化了音频信号处理流程，成功构建了一套实时音频信号处理系统。特征提取是语音信号处理的基础步骤，梅尔频率倒谱系数（MFCCs）因其与人类听觉感知的高度相关性而成为主流选择。然而传统MFCC提取依赖计算密集的时频变换，限制了实时应用效率。为此，我们提出利用储备计算优化MFCC提取的新方案——通过用卷积运算替代传统频域转换，在保持特征判别力的同时消除了复杂变换需求。我们展示了集成该方法的端到端音频处理框架，验证了其在高效实时语音分析中的潜力。本研究结果推动了节能型音频处理技术的发展，为嵌入式系统及语音驱动应用的无缝部署提供了可能。这项工作弥合了生物启发特征提取与现代神经形态计算之间的鸿沟，为下一代语音识别系统提供了可扩展的解决方案。
URL
https://arxiv.org/abs/2603.24283
PDF
-
From Oracle to Noisy Context: Mitigating Contextual Exposure Bias in Speech-LLMs
2026-03-25 07:48:04Xiaoyong Guo, Nanjie Li, Zijie Zeng, Kai Wang, Hao Huang, Haihua Xu, Wei ShiAbstract
Contextual automatic speech recognition (ASR) with Speech-LLMs is typically trained with oracle conversation history, but relies on error-prone history at inference, causing a train-test mismatch in the context channel that we term contextual exposure bias. We propose a unified training framework to improve robustness under realistic histories: (i) Teacher Error Knowledge by using Whisper large-v3 hypotheses as training-time history, (ii) Context Dropout to regularize over-reliance on history, and (iii) Direct Preference Optimization (DPO) on curated failure cases. Experiments on TED-LIUM 3 (in-domain) and zero-shot LibriSpeech (out-of-domain) show consistent gains under predicted-history decoding. With a two-utterance history as context, SFT with Whisper hypotheses reduce WER from 5.59% (oracle-history training) to 5.47%, and DPO further improves to 5.17%. Under irrelevant-context attacks, DPO yields the smallest degradation (5.17% -> 5.63%), indicating improved robustness to misleading context. Our code and models are published on this https URL.
Abstract (translated)
基于语音大语言模型（Speech-LLM）的上下文自动语音识别（ASR）通常使用真实对话历史（oracle history）进行训练，但在推理时却依赖于易出错的预测历史，导致上下文通道中存在训练-测试不匹配问题，我们称之为上下文暴露偏差（contextual exposure bias）。我们提出一个统一训练框架以提升模型在真实历史条件下的鲁棒性：（i）教师错误知识（Teacher Error Knowledge），即使用Whisper large-v3的假设文本作为训练时的历史输入；（ii）上下文丢弃（Context Dropout），用于正则化模型对历史的过度依赖；（iii）对精选的失败案例进行直接偏好优化（DPO）。在TED-LIUM 3（领域内）和零样本LibriSpeech（领域外）上的实验表明，在预测历史解码条件下该方法持续取得增益。当使用两句历史作为上下文时，基于Whisper假设进行监督微调（SFT）可将词错误率（WER）从5.59%（真实历史训练）降至5.47%，而DPO进一步优化至5.17%。在无关上下文攻击场景下，DPO产生的性能退化最小（5.17% → 5.63%），表明其对误导性上下文具有更强的鲁棒性。代码与模型已公开发布于此https链接。
URL
https://arxiv.org/abs/2603.24034
PDF
-
Ethio-ASR: Joint Multilingual Speech Recognition and Language Identification for Ethiopian Languages
2026-03-24 18:55:45Badr M. Abdullah, Israel Abebe Azime, Atnafu Lambebo Tonja, Jesujoba O. Alabi, Abel Mulat Alemu, Eyob G. Hagos, Bontu Fufa Balcha, Mulubrhan A. Nerea, Debela Desalegn Yadeta, Dagnachew Mekonnen Marilign, Amanuel Temesgen Fentahun, Tadesse Kebede, Israel D. Gebru, Michael Melese Woldeyohannis, Walelign Tewabe Sewunetie, Bernd M\"obius, Dietrich KlakowAbstract
We present Ethio-ASR, a suite of multilingual CTC-based automatic speech recognition (ASR) models jointly trained on five Ethiopian languages: Amharic, Tigrinya, Oromo, Sidaama, and Wolaytta. These languages belong to the Semitic, Cushitic, and Omotic branches of the Afroasiatic family, and remain severely underrepresented in speech technology despite being spoken by the vast majority of Ethiopia's population. We train our models on the recently released WAXAL corpus using several pre-trained speech encoders and evaluate against strong multilingual baselines, including OmniASR. Our best model achieves an average WER of 30.48% on the WAXAL test set, outperforming the best OmniASR model with substantially fewer parameters. We further provide a comprehensive analysis of gender bias, the contribution of vowel length and consonant gemination to ASR errors, and the training dynamics of multilingual CTC models. Our models and codebase are publicly available to the research community.
Abstract (translated)
我们推出了Ethio-ASR，这是一套基于CTC的多语言自动语音识别模型，在埃塞俄比亚五种语言（阿姆哈拉语、提格雷语、奥罗莫语、西达马语和沃莱塔语）上进行了联合训练。这些语言属于亚非语系的闪米特语支、库希特语支和奥莫特语支，尽管被埃塞俄比亚绝大多数人口使用，但在语音技术领域仍严重缺乏代表性。我们基于近期发布的WAXAL语料库训练模型，采用了多种预训练语音编码器，并与OmniASR等强大多语言基线模型进行对比评估。最佳模型在WAXAL测试集上达到平均30.48%的词错误率，以显著更少的参数数量超越了最优的OmniASR模型。我们还进一步提供了关于性别偏差、元音长度与辅音重叠对识别错误的影响，以及多语言CTC模型训练动态的全面分析。我们的模型和代码库已向研究社区公开。
URL
https://arxiv.org/abs/2603.23654
PDF
-
Evaluating a Multi-Agent Voice-Enabled Smart Speaker for Care Homes: A Safety-Focused Framework
2026-03-24 18:08:06Zeinab Dehghani, Rameez Raja Kureshi, Koorosh Aslansefat, Faezeh Alsadat Abedi, Dhavalkumar Thakker, Lisa Greaves, Bhupesh Kumar Mishra, Baseer Ahmad, Tanaya MaslekarAbstract
Artificial intelligence (AI) is increasingly being explored in health and social care to reduce administrative workload and allow staff to spend more time on patient care. This paper evaluates a voice-enabled Care Home Smart Speaker designed to support everyday activities in residential care homes, including spoken access to resident records, reminders, and scheduling tasks. A safety-focused evaluation framework is presented that examines the system end-to-end, combining Whisper-based speech recognition with retrieval-augmented generation (RAG) approaches (hybrid, sparse, and dense). Using supervised care-home trials and controlled testing, we evaluated 330 spoken transcripts across 11 care categories, including 184 reminder-containing interactions. These evaluations focus on (i) correct identification of residents and care categories, (ii) reminder recognition and extraction, and (iii) end-to-end scheduling correctness under uncertainty (including safe deferral/clarification). Given the safety-critical nature of care homes, particular attention is also paid to reliability in noisy environments and across diverse accents, supported by confidence scoring, clarification prompts, and human-in-the-loop oversight. In the best-performing configuration (GPT-5.2), resident ID and care category matching reached 100% (95% CI: 98.86-100), while reminder recognition reached 89.09\% (95% CI: 83.81-92.80) with zero missed reminders (100% recall) but some false positives. End-to-end scheduling via calendar integration achieved 84.65% exact reminder-count agreement (95% CI: 78.00-89.56), indicating remaining edge cases in converting informal spoken instructions into actionable events. The findings suggest that voice-enabled systems, when carefully evaluated and appropriately safeguarded, can support accurate documentation, effective task management, and trustworthy use of AI in care home settings.
Abstract (translated)
人工智能（AI）在健康与社会照护领域的应用日益受到关注，旨在减轻行政工作负担，使工作人员能更专注于患者照护。本文评估了一款面向养老院日常活动的语音智能音箱系统，该系统支持通过语音访问居民记录、提醒事项及任务调度。研究提出了一种以安全为核心的端到端评估框架，结合基于Whisper的语音识别与检索增强生成（RAG）方法（包括混合、稀疏和密集模式）。通过监督式养老院试验与受控测试，我们对11个照护类别中的330条语音转录文本进行了评估，其中包含184条带提醒内容的交互。评估聚焦于：（i）居民身份与照护类别的正确识别；（ii）提醒事项的识别与提取；（iii）不确定环境下端到端调度的正确性（包括安全延迟/澄清机制）。鉴于养老院场景对安全性的高要求，研究特别关注系统在嘈杂环境及多样口音下的可靠性，并通过置信度评分、澄清提示及人在环中监督机制提供支持。在表现最佳的配置（GPT-5.2）中，居民ID与照护类别匹配准确率达100%（95%置信区间：98.86-100），提醒识别率达89.09%（95%置信区间：83.81-92.80），实现零遗漏提醒（100%召回率），但存在部分误报。通过日历集成实现的端到端调度，其提醒数量精确匹配率为84.65%（95%置信区间：78.00-89.56），表明将非正式语音指令转化为可执行事件仍存在边缘案例。研究结果表明，经过严格评估并配备适当防护措施的语音系统，能够在养老院场景中支持准确记录、高效任务管理以及值得信赖的AI应用。
URL
https://arxiv.org/abs/2603.23625
PDF
-
From Content to Audience: A Multimodal Annotation Framework for Broadcast Television Analytics
2026-03-24 13:46:51Paolo Cupini, Francesco PierriAbstract
Automated semantic annotation of broadcast television content presents distinctive challenges, combining structured audiovisual composition, domain-specific editorial patterns, and strict operational constraints. While multimodal large language models (MLLMs) have demonstrated strong general-purpose video understanding capabilities, their comparative effectiveness across pipeline architectures and input configurations in broadcast-specific settings remains empirically undercharacterized. This paper presents a systematic evaluation of multimodal annotation pipelines applied to broadcast television news in the Italian setting. We construct a domain-specific benchmark of clips labeled across four semantic dimensions: visual environment classification, topic classification, sensitive content detection, and named entity recognition. Two different pipeline architectures are evaluated across nine frontier models, including Gemini 3.0 Pro, LLaMA 4 Maverick, Qwen-VL variants, and Gemma 3, under progressively enriched input strategies combining visual signals, automatic speech recognition, speaker diarization, and metadata. Experimental results demonstrate that gains from video input are strongly model-dependent: larger models effectively leverage temporal continuity, while smaller models show performance degradation under extended multimodal context, likely due to token overload. Beyond benchmarking, the selected pipeline is deployed on 14 full broadcast episodes, with minute-level annotations integrated with normalized audience measurement data provided by an Italian media company. This integration enables correlational analysis of topic-level audience sensitivity and generational engagement divergence, demonstrating the operational viability of the proposed framework for content-based audience analytics.
Abstract (translated)
广播电视内容的自动化语义标注面临独特挑战，需综合处理结构化的视听构成、领域特定的编辑模式以及严格的操作限制。尽管多模态大语言模型（MLLMs）已展现出强大的通用视频理解能力，但针对广播电视特定场景下不同流水线架构与输入配置的相对有效性，尚缺乏实证研究。本文系统评估了应用于意大利语境下电视新闻的多模态标注流水线。我们构建了涵盖四个语义维度的领域专用基准数据集：视觉环境分类、话题分类、敏感内容检测与命名实体识别。实验在九种前沿模型（包括Gemini 3.0 Pro、LLaMA 4 Maverick、Qwen-VL系列及Gemma 3）上对比了两种流水线架构，并采用渐进式增强的输入策略，逐步融合视觉信号、自动语音识别、说话人分离与元数据。实验结果表明，视频输入带来的性能提升高度依赖模型规模：大模型能有效利用时间连续性，而小模型在扩展多模态上下文时出现性能退化，可能源于token过载。除基准测试外，选定流水线已部署于14集完整广播节目，其分钟级标注与意大利媒体公司提供的标准化收视率数据实现整合。该整合支持了话题级受众敏感度与代际参与差异的相关性分析，验证了所提框架在基于内容的受众分析中的 operational viability。
URL
https://arxiv.org/abs/2603.26772
PDF
-
MSR-HuBERT: Self-supervised Pre-training for Adaptation to Multiple Sampling Rates
2026-03-24 10:40:23Zikang Huang, Meng Ge, Tianrui Wang, Xuanchen Li, Xiaobao Wang, Longbiao Wang, Jianwu DangAbstract
Self-supervised learning (SSL) has advanced speech processing. However, existing speech SSL methods typically assume a single sampling rate and struggle with mixed-rate data due to temporal resolution mismatch. To address this limitation, we propose MSRHuBERT, a multi-sampling-rate adaptive pre-training method. Building on HuBERT, we replace its single-rate downsampling CNN with a multi-sampling-rate adaptive downsampling CNN that maps raw waveforms from different sampling rates to a shared temporal resolution without resampling. This design enables unified mixed-rate pre-training and fine-tuning. In experiments spanning 16 to 48 kHz, MSRHuBERT outperforms HuBERT on speech recognition and full-band speech reconstruction, preserving high-frequency detail while modeling low-frequency semantic structure. Moreover, MSRHuBERT retains HuBERT's mask-prediction objective and Transformer encoder, so existing analyses and improvements that were developed for HuBERT can apply directly.
Abstract (translated)
自监督学习（SSL）推动了语音处理的发展。然而，现有的语音SSL方法通常假设单一采样率，并因时间分辨率不匹配而难以处理混合采样率数据。为应对这一局限，我们提出了MSRHuBERT，一种多采样率自适应预训练方法。该方法在HuBERT基础上，将其单采样率下采样CNN替换为多采样率自适应下采样CNN，可将不同采样率的原始波形映射至共享时间分辨率，无需重采样。此设计支持统一的混合采样率预训练与微调。在16至48kHz范围的实验中，MSRHuBERT在语音识别与全频带语音重建任务上均优于HuBERT，既保留高频细节，又能对低频语义结构进行建模。此外，MSRHuBERT保留了HuBERT的掩码预测目标与Transformer编码器，因此此前为HuBERT开发的分析方法与改进方案可直接适用。
URL
https://arxiv.org/abs/2603.23048
PDF
-
When AVSR Meets Video Conferencing: Dataset, Degradation, and the Hidden Mechanism Behind Performance Collapse
2026-03-24 08:03:58Yihuan Huang, Jun Xue, Liu Jiajun, Daixian Li, Tong Zhang, Zhuolin Yi, Yanzhen Ren, Kai LiAbstract
Audio-Visual Speech Recognition (AVSR) has achieved remarkable progress in offline conditions, yet its robustness in real-world video conferencing (VC) remains largely unexplored. This paper presents the first systematic evaluation of state-of-the-art AVSR models across mainstream VC platforms, revealing severe performance degradation caused by transmission distortions and spontaneous human hyper-expression. To address this gap, we construct \textbf{MLD-VC}, the first multimodal dataset tailored for VC, comprising 31 speakers, 22.79 hours of audio-visual data, and explicit use of the Lombard effect to enhance human hyper-expression. Through comprehensive analysis, we find that speech enhancement algorithms are the primary source of distribution shift, which alters the first and second formants of audio. Interestingly, we find that the distribution shift induced by the Lombard effect closely resembles that introduced by speech enhancement, which explains why models trained on Lombard data exhibit greater robustness in VC. Fine-tuning AVSR models on MLD-VC mitigates this issue, achieving an average 17.5% reduction in CER across several VC platforms. Our findings and dataset provide a foundation for developing more robust and generalizable AVSR systems in real-world video conferencing. MLD-VC is available at this https URL.
Abstract (translated)
音视频语音识别（AVSR）在离线条件下已取得显著进展，但其在真实视频会议（VC）场景中的鲁棒性尚缺乏系统研究。本文首次对主流VC平台上的先进AVSR模型进行了系统评估，揭示了传输失真与人类夸张自发表达导致的严重性能下降。为填补此空白，我们构建了首个面向视频会议场景的多模态数据集\textbf{MLD-VC}，包含31位说话人、22.79小时音视频数据，并明确引入伦巴德效应以增强人类夸张表达。通过全面分析发现，语音增强算法是导致分布偏移的主要来源，该偏移改变了音频的一、二阶共振峰。有趣的是，伦巴德效应引发的分布偏移与语音增强引入的偏移高度相似，这解释了为何在伦巴德数据上训练的模型在视频会议中表现出更强鲁棒性。基于MLD-VC对AVSR模型进行微调可缓解此问题，在多个VC平台上平均降低17.5%的字错误率。本研究结果与数据集为开发更鲁棒、泛化能力更强的真实视频会议音视频语音识别系统奠定了基础。MLD-VC数据集可通过此超链接获取。
URL
https://arxiv.org/abs/2603.22915
PDF
-
Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics
2026-03-24 02:01:21Naohiro Tawara, Samuele Cornell, Alexander Polok, Marc Delcroix, Luk\'a\v{s} Burget, Shinji WatanabeAbstract
Conversational automatic speech recognition remains challenging due to overlapping speech, far-field noise, and varying speaker counts. While recent LLM-based systems perform well on single-speaker benchmarks, their robustness in multi-speaker settings is unclear. We systematically compare LLM-based and modular pipeline approaches along four axes: overlap robustness, semantic fidelity, speaker count, and single- versus multi-channel input. To capture meaning-altering errors that conventional metrics miss, we introduce tcpSemER, which extends tcpWER by replacing Levenshtein distance with embedding-based semantic similarity. We further decompose tcpWER into overlapping and non-overlapping components for finer-grained analysis. Experiments across three datasets show that LLM-based systems are competitive in two-speaker settings but degrade as speaker count and overlap increase, whereas modular pipelines remain more robust.
Abstract (translated)
对话式自动语音识别仍面临重叠语音、远场噪声和说话人数变化等挑战。尽管近期基于大语言模型的系统在单说话人基准测试中表现良好，但其在多说话人场景下的鲁棒性尚不明确。我们从四个维度系统比较了基于大语言模型与模块化流水线方法：重叠鲁棒性、语义保真度、说话人数以及单通道与多通道输入。为捕捉传统指标遗漏的语义改变错误，我们提出了tcpSemER，该指标通过用基于嵌入的语义相似度替代编辑距离来扩展tcpWER。我们还进一步将tcpWER分解为重叠与非重叠部分，以实现更细粒度的分析。在三个数据集上的实验表明，基于大语言模型的系统在双说话人场景中具有竞争力，但随着说话人数增加和重叠加剧，其性能下降，而模块化流水线方法则保持更高鲁棒性。
URL
https://arxiv.org/abs/2603.22709
PDF
-
SeaAlert: Critical Information Extraction From Maritime Distress Communications with Large Language Models
2026-03-23 18:21:13Tomer Atia, Yehudit Aperstein, Alexander ApartsinAbstract
Maritime distress communications transmitted over very high frequency (VHF) radio are safety-critical voice messages used to report emergencies at sea. Under the Global Maritime Distress and Safety System (GMDSS), such messages follow standardized procedures and are expected to convey essential details, including vessel identity, position, nature of the distress, and required assistance. In practice, however, automatic analysis remains difficult because distress messages are often brief, noisy, and produced under stress, may deviate from the prescribed format, and are further degraded by automatic speech recognition (ASR) errors caused by channel noise and speaker stress. This paper presents SeaAlert, an LLM-based framework for robust analysis of maritime distress communications. To address the scarcity of labeled real-world data, we develop a synthetic data generation pipeline in which an LLM produces realistic and diverse maritime messages, including challenging variants in which standard distress codewords are omitted or replaced with less explicit expressions. The generated utterances are synthesized into speech, degraded with simulated VHF noise, and transcribed by an ASR system to obtain realistic noisy transcripts.
Abstract (translated)
URL
https://arxiv.org/abs/2604.14163
PDF
-
SLURP-TN : Resource for Tunisian Dialect Spoken Language Understanding
2026-03-23 12:59:11Haroun Elleuch, Salima Mdhaffar, Yannick Est\`eve, Fethi BougaresAbstract
Spoken Language Understanding (SLU) aims to extract the semantic information from the speech utterance of user queries. It is a core component in a task-oriented dialogue system. With the spectacular progress of deep neural network models and the evolution of pre-trained language models, SLU has obtained significant breakthroughs. However, only a few high-resource languages have taken advantage of this progress due to the absence of SLU resources. In this paper, we seek to mitigate this obstacle by introducing SLURP-TN. This dataset was created by recording 55 native speakers uttering sentences in Tunisian dialect, manually translated from six SLURP domains. The result is an SLU Tunisian dialect dataset that comprises 4165 sentences recorded into around 5 hours of acoustic material. We also develop a number of Automatic Speech Recognition and SLU models exploiting SLUTP-TN. The Dataset and baseline models are available at: this https URL.
Abstract (translated)
口语理解（SLU）旨在从用户查询的语音表述中提取语义信息，是任务型对话系统的核心组件。随着深度神经网络模型的显著进步和预训练语言模型的演进，SLU已取得重大突破。然而，由于SLU资源的匮乏，仅有少数高资源语言得以受益于此进展。本文通过引入SLURP-TN数据集来缓解这一障碍：该数据集由55名母语者录制完成，内容源自SLURP数据集中六个领域的句子手工翻译而成的突尼斯方言。最终构建的SLU突尼斯方言数据集包含4165句录音，约5小时声学材料。我们还开发了一系列基于SLURP-TN的自动语音识别与SLU模型。数据集及基线模型已开源，链接如下：此https URL。
URL
https://arxiv.org/abs/2603.21940
PDF
-
Ara-Best-RQ: Multi Dialectal Arabic SSL
2026-03-23 12:23:31Haroun Elleuch, Ryan Whetten, Salima Mdhaffar, Yannick Est\`eve, Fethi BougaresAbstract
We present Ara-BEST-RQ, a family of self-supervised learning (SSL) models specifically designed for multi-dialectal Arabic speech processing. Leveraging 5,640 hours of crawled Creative Commons speech and combining it with publicly available datasets, we pre-train conformer-based BEST-RQ models up to 600M parameters. Our models are evaluated on dialect identification (DID) and automatic speech recognition (ASR) tasks, achieving state-of-the-art performance on the former while using fewer parameters than competing models. We demonstrate that family-targeted pre-training on Arabic dialects significantly improves downstream performance compared to multilingual or monolingual models trained on non-Arabic data. All models, code, and pre-processed datasets will be publicly released to support reproducibility and further research in Arabic speech technologies.
Abstract (translated)
我们提出了Ara-BEST-RQ，这是一系列专为多方言阿拉伯语语音处理设计的自监督学习（SSL）模型。通过利用5,640小时爬取的创作共用语音数据，并结合公开数据集，我们预训练了基于Conformer架构的BEST-RQ模型，参数量最高达6亿。我们的模型在方言识别（DID）和自动语音识别（ASR）任务上进行了评估，在前者任务上达到了最先进水平，且参数量少于竞争模型。我们证明，针对阿拉伯语方言进行的家族式预训练，相较于使用非阿拉伯语数据训练的多语言或单语模型，能显著提升下游任务性能。所有模型、代码及预处理数据集将公开释放，以支持阿拉伯语语音技术的可复现性与进一步研究。
URL
https://arxiv.org/abs/2603.21900
PDF
-
Cascade-Free Mandarin Visual Speech Recognition via Semantic-Guided Cross-Representation Alignment
2026-03-23 10:52:02Lei Yang, Yi He, Fei Wu, Shilin WangAbstract
Chinese mandarin visual speech recognition (VSR) is a task that has advanced in recent years, yet still lags behind the performance on non-tonal languages such as English. One primary challenge arises from the tonal nature of Mandarin, which limits the effectiveness of conventional sequence-to-sequence modeling approaches. To alleviate this issue, existing Chinese VSR systems commonly incorporate intermediate representations, most notably pinyin, within cascade architectures to enhance recognition accuracy. While beneficial, in these cascaded designs, the subsequent stage during inference depends on the output of the preceding stage, leading to error accumulation and increased inference latency. To address these limitations, we propose a cascade-free architecture based on multitask learning that jointly integrates multiple intermediate representations, including phoneme and viseme, to better exploit contextual information. The proposed semantic-guided local contrastive loss temporally aligns the features, enabling on-demand activation during inference, thereby providing a trade-off between inference efficiency and performance while mitigating error accumulation caused by projection and re-embedding. Experiments conducted on publicly available datasets demonstrate that our method achieves superior recognition performance.
Abstract (translated)
URL
https://arxiv.org/abs/2603.21808
PDF
-
Demonstration of Adapt4Me: An Uncertainty-Aware Authoring Environment for Personalizing Automatic Speech Recognition to Non-normative Speech
2026-03-20 16:36:44Niclas Pokel, Yiming Zhao, Pehu\'en Moure, Yingqiang Gao, Roman B\"ohringerAbstract
Personalizing Automatic Speech Recognition (ASR) for non-normative speech remains challenging because data collection is labor-intensive and model training is technically complex. To address these limitations, we propose Adapt4Me, a web-based decentralized environment that operationalizes Bayesian active learning to enable end-to-end personalization without expert supervision. The app exposes data selection, adaptation, and validation to lay users through a three-stage human-in-the-loop workflow: (1) rapid profiling via greedy phoneme sampling to capture speaker-specific acoustics; (2) backend personalization using Variational Inference Low-Rank Adaptation (VI-LoRA) to enable fast, incremental updates; and (3) continuous improvement, where users guide model refinement by resolving visualized model uncertainty via low-friction top-k corrections. By making epistemic uncertainty explicit, Adapt4Me reframes data efficiency as an interactive design feature rather than a purely algorithmic concern. We show how this enables users to personalize robust ASR models, transforming them from passive data sources into active authors of their own assistive technology.
Abstract (translated)
针对非规范语音的个性化自动语音识别（ASR）仍面临挑战，因为数据收集耗时费力，模型训练技术复杂。为突破这些限制，我们提出了Adapt4Me——一个基于网络的去中心化环境，它将贝叶斯主动学习落地为可操作方案，实现无需专家监督的端到端个性化。该应用通过三阶段人机协同工作流，将数据选择、适配与验证向普通用户开放：(1) 通过贪婪音素采样进行快速特征提取，捕捉说话人特有的声学信息；(2) 后端采用变分推断低秩适配（VI-LoRA）实现快速增量更新；(3) 通过低摩擦度的top-k修正机制，让用户通过解决可视化模型不确定性来持续优化模型。Adapt4Me通过将认知不确定性显性化，将数据效率重新定义为交互式设计特性，而非纯算法问题。我们展示了这一方案如何让用户个性化鲁棒ASR模型，使他们从被动数据提供者转变为自身辅助技术的主动创造者。
URL
https://arxiv.org/abs/2603.20112
PDF
