Abstract
Advances in large language models (LLMs) have enabled significant capabilities in audio processing, resulting in state-of-the-art models now known as Large Audio Language Models (LALMs). However, minimal work has been done to measure audio understanding beyond automatic speech recognition (ASR). This paper closes that gap by proposing a benchmark suite, SCENEBench (Spatial, Cross-lingual, Environmental, Non-speech Evaluation), that targets a broad form of audio comprehension across four real-world categories: background sound understanding, noise localization, cross-linguistic speech understanding, and vocal characterizer recognition. These four categories are selected based on understudied needs from accessibility technology and industrial noise monitoring. In addition to performance, we also measure model latency. The purpose of this benchmark suite is to assess audio beyond just what words are said - rather, how they are said and the non-speech components of the audio. Because our audio samples are synthetically constructed (e.g., by overlaying two natural audio samples), we further validate our benchmark against 20 natural audio items per task, sub-sampled from existing datasets to match our task criteria, to assess ecological validity. We assess five state-of-the-art LALMs and find critical gaps: performance varies across tasks, with some tasks performing below random chance and others achieving high accuracy. These results provide direction for targeted improvements in model capabilities.
Abstract (translated)
大型语言模型（LLM）的进步使音频处理能力得到了显著提升，进而产生了被称为大音频语言模型（LALM）的先进模型。然而，在自动语音识别（ASR）之外对音频理解进行度量的工作仍然很少。本文通过提出一个基准测试套件SCENEBench来填补这一空白，该套件涵盖了四大类现实世界中的音频理解：背景声音理解、噪音定位、跨语言口语理解和声源特征识别。这四个类别是基于无障碍技术和工业噪声监测领域中研究不足的需求而选定的。 除了性能之外，我们还对模型延迟进行了测量。此基准测试套件旨在评估不仅仅是说出了什么词，更重要的是如何说出这些话以及音频中的非言语部分。由于我们的音频样本是由合成方法构建的（例如，在自然音频样本上叠加），为了验证生态有效性，我们在每个任务中选取了20个从现有数据集中随机抽样的自然音频样本来与合成测试进行对比。 我们评估了五个最先进的LALM，并发现了重要的差距：在各个任务中的性能表现不一，有些任务的表现甚至低于随机猜测的水平，而其他任务则能达到较高的准确性。这些结果为模型能力的具体改进指明了方向。
URL
https://arxiv.org/abs/2603.09853
