Abstract
Deriving operational intelligence from organizational data repositories is a key challenge due to the dichotomy of data confidentiality vs data integration objectives, as well as the limitations of Natural Language Processing (NLP) tools relative to the specific knowledge structure of domains such as operations and maintenance. In this work, we discuss Knowledge Graph construction and break down the Knowledge Extraction process into its Named Entity Recognition, Coreference Resolution, Named Entity Linking, and Relation Extraction functional components. We then evaluate sixteen NLP tools in concert with or in comparison to the rapidly advancing capabilities of Large Language Models (LLMs). We focus on the operational and maintenance intelligence use case for trusted applications in the aircraft industry. A baseline dataset is derived from a rich public domain US Federal Aviation Administration dataset focused on equipment failures or maintenance requirements. We assess the zero-shot performance of NLP and LLM tools that can be operated within a controlled, confidential environment (no data is sent to third parties). Based on our observation of significant performance limitations, we discuss the challenges related to trusted NLP and LLM tools as well as their Technical Readiness Level for wider use in mission-critical industries such as aviation. We conclude with recommendations to enhance trust and provide our open-source curated dataset to support further baseline testing and evaluation.
Abstract (translated)
从组织数据仓库中提取运营智能是一项关键挑战，这主要是由于数据保密性和数据集成目标之间的矛盾，以及自然语言处理（NLP）工具相对于诸如操作和维护等领域特定知识结构的限制。在这项工作中，我们讨论了知识图谱构建，并将知识抽取过程分解为命名实体识别、共指解析、命名实体链接和关系提取等功能组件。然后，我们将16种NLP工具与大型语言模型（LLM）迅速发展的能力进行对比评估。我们的研究重点是航空业受信任应用中的运营和维护智能案例。基准数据集是从美国联邦航空管理局丰富的公共领域数据集中衍生出来的，该数据集专注于设备故障或维修需求。我们评估了在可控、保密环境中运行的NLP和LLM工具的零样本性能（不向第三方发送任何数据）。鉴于显著的性能限制，我们讨论了受信任的NLP和LLM工具面临的挑战及其在航空等关键任务行业更广泛应用的技术成熟度水平。最后，我们提出增强信任的建议，并提供我们的开源精选数据集以支持进一步的基础测试和评估。
URL
https://arxiv.org/abs/2507.22935
