Abstract
Despite rapid advances in large language models (LLMs), their linguistic abilities in low-resource and morphologically rich languages are still not well understood due to limited annotated resources and the absence of standardized evaluation frameworks. This paper presents LLM Probe, a lexicon-based assessment framework designed to systematically evaluate the linguistic skills of LLMs in low-resource language environments. The framework analyzes models across four areas of language understanding: lexical alignment, part-of-speech recognition, morphosyntactic probing, and translation accuracy. To illustrate the framework, we create a manually annotated benchmark dataset using a low-resource Semitic language as a case study. The dataset comprises bilingual lexicons with linguistic annotations, including part-of-speech tags, grammatical gender, and morphosyntactic features, which demonstrate high inter-annotator agreement to ensure reliable annotations. We test a variety of models, including causal language models and sequence-to-sequence architectures. The results reveal notable differences in performance across various linguistic tasks: sequence-to-sequence models generally excel in morphosyntactic analysis and translation quality, whereas causal models demonstrate strong performance in lexical alignment but exhibit weaker translation accuracy. Our results emphasize the need for linguistically grounded evaluation to better understand LLM limitations in low-resource settings. We release LLM Probe and the accompanying benchmark dataset as open-source tools to promote reproducible benchmarking and to support the development of more inclusive multilingual language technologies.
Abstract (translated)
尽管大语言模型（LLMs）发展迅速，但由于标注资源有限且缺乏标准化评估框架，其在低资源及形态丰富语言中的语言能力仍未被充分理解。本文提出LLM Probe——一个基于词典的评估框架，旨在系统化评估LLMs在低资源语言环境中的语言技能。该框架从四个语言理解维度分析模型表现：词汇对齐、词性识别、形态句法探测及翻译准确性。为展示框架应用，我们以低资源闪米特语为案例，构建了手动标注的基准数据集。该数据集包含带语言注释的双语词典，涵盖词性标签、语法性属及形态句法特征，并具有较高的标注者间一致性以确保标注可靠性。我们测试了包括因果语言模型与序列到序列架构在内的多种模型。结果显示不同语言任务间存在显著性能差异：序列到序列模型通常在形态句法分析与翻译质量上表现更优，而因果模型在词汇对齐方面表现强劲但翻译准确性较弱。研究结果强调需要基于语言学基础的评估以更好理解LLMs在低资源场景下的局限性。我们将LLM Probe及配套基准数据集作为开源工具发布，以促进可复现的基准测试，并支持更具包容性的多语言语言技术发展。
URL
https://arxiv.org/abs/2603.29517
