Abstract
Evaluating ASR systems for Indian languages is challenging due to spelling variations, suffix splitting flexibility, and non-standard spellings in code-mixed words. Traditional Word Error Rate (WER) often presents a bleaker picture of system performance than what human users perceive. Better aligning evaluation with real-world performance requires capturing permissible orthographic variations, which is extremely challenging for under-resourced Indian languages. Leveraging recent advances in LLMs, we propose a framework for creating benchmarks that capture permissible variations. Through extensive experiments, we demonstrate that OIWER, by accounting for orthographic variations, reduces pessimistic error rates (an average improvement of 6.3 points), narrows inflated model gaps (e.g., Gemini-Canary performance difference drops from 18.1 to 11.5 points), and aligns more closely with human perception than prior methods like WER-SN by 4.9 points.
Abstract (translated)
评估印度语言的自动语音识别（ASR）系统具有挑战性，原因包括拼写变化、词尾分割灵活性以及代码混合单词中的非标准拼写。传统的词错误率（WER）常常比人类用户的实际体验更为悲观地描绘系统的性能。为了更好地将评估与现实世界的性能相匹配，需要捕捉允许的正字法变化，这对资源匮乏的印度语言来说尤其具有挑战性。借助最近在大型语言模型（LLM）领域的进展，我们提出了一种框架来创建能够捕捉允许的变化的基准测试。通过广泛的实验，我们证明了考虑到正字法变化的对象初始化词错误率（OIWER），减少了悲观的误差率（平均提高了6.3个百分点）、缩小了夸大后的模型差距（例如，Gemini-Canary性能差异从18.1降至11.5个百分点）并且比之前的评估方法如WER-SN更紧密地与人类感知对齐（高出4.9个百分点）。
URL
https://arxiv.org/abs/2603.00941
