Abstract
Generative AI (GEN AI) models have revolutionized diverse application domains but present substantial challenges due to reliability concerns, including hallucinations, semantic drift, and inherent biases. These models typically operate as black-boxes, complicating transparent and objective evaluation. Current evaluation methods primarily depend on subjective human assessment, limiting scalability, transparency, and effectiveness. This research proposes a systematic methodology using deterministic and Large Language Model (LLM)-generated Knowledge Graphs (KGs) to continuously monitor and evaluate GEN AI reliability. We construct two parallel KGs: (i) a deterministic KG built using explicit rule-based methods, predefined ontologies, domain-specific dictionaries, and structured entity-relation extraction rules, and (ii) an LLM-generated KG dynamically derived from real-time textual data streams such as live news articles. Utilizing real-time news streams ensures authenticity, mitigates biases from repetitive training, and prevents adaptive LLMs from bypassing predefined benchmarks through feedback memorization. To quantify structural deviations and semantic discrepancies, we employ several established KG metrics, including Instantiated Class Ratio (ICR), Instantiated Property Ratio (IPR), and Class Instantiation (CI). An automated real-time monitoring framework continuously computes deviations between deterministic and LLM-generated KGs. By establishing dynamic anomaly thresholds based on historical structural metric distributions, our method proactively identifies and flags significant deviations, thus promptly detecting semantic anomalies or hallucinations. This structured, metric-driven comparison between deterministic and dynamically generated KGs delivers a robust and scalable evaluation framework.
Abstract (translated)
生成式人工智能（GEN AI）模型已在多个应用领域实现了革命性的变革，但它们由于可靠性问题而面临重大挑战，这些问题包括幻觉、语义漂移和固有偏见。这些模型通常以黑箱形式运作，这使得透明且客观的评估变得复杂化。当前的评估方法主要依赖于主观的人类评判，从而限制了其可扩展性、透明度和有效性。本研究提出了一种系统的方法，利用确定性的知识图谱（KG）和大型语言模型（LLM）生成的知识图谱来持续监控和评估GEN AI的可靠性。我们构建了两个平行的知识图谱：(i) 一个使用明确规则基于方法、预定义的本体论、领域特定字典以及结构化的实体-关系提取规则构建的确定性知识图谱，(ii) 一个从实时文本流（如即时新闻文章）中动态生成的LLM知识图谱。利用实时新闻流确保了真实性，并且可以缓解重复训练导致的偏见问题，防止通过反馈记忆使适应性强的LLM绕过预定义基准的问题。 为了量化结构偏差和语义差异，我们采用了几种已建立的知识图谱指标，包括实例化类比例（ICR）、实例化属性比例（IPR）以及类实例化（CI）。一个自动化的实时监控框架持续计算确定性知识图谱与LLM生成的知识图谱之间的偏差。通过基于历史结构指标分布来动态设置异常阈值，我们的方法能够提前识别和标记重要的差异，并及时检测语义异常或幻觉。 这种基于结构化、度量驱动的方法对确定性和动态生成的知识图谱进行比较，为评估提供了强大且可扩展的框架。
URL
https://arxiv.org/abs/2509.03857
