Abstract
Medical domain automated text generation is an active area of research and development; however, evaluating the clinical quality of generated reports remains a challenge, especially in instances where domain-specific metrics are lacking, e.g. histopathology. We propose HARE (Histopathology Automated Report Evaluation), a novel entity and relation centric framework, composed of a benchmark dataset, a named entity recognition (NER) model, a relation extraction (RE) model, and a novel metric, which prioritizes clinically relevant content by aligning critical histopathology entities and relations between reference and generated reports. To develop the HARE benchmark, we annotated 813 de-identified clinical diagnostic histopathology reports and 652 histopathology reports from The Cancer Genome Atlas (TCGA) with domain-specific entities and relations. We fine-tuned GatorTronS, a domain-adapted language model to develop HARE-NER and HARE-RE which achieved the highest overall F1-score (0.915) among the tested models. The proposed HARE metric outperformed traditional metrics including ROUGE and Meteor, as well as radiology metrics such as RadGraph-XL, with the highest correlation and the best regression to expert evaluations (higher than the second best method, GREEN, a large language model based radiology report evaluator, by Pearson $r = 0.168$, Spearman $\rho = 0.161$, Kendall $\tau = 0.123$, $R^2 = 0.176$, $RMSE = 0.018$). We release HARE, datasets, and the models at this https URL to foster advancements in histopathology report generation, providing a robust framework for improving the quality of reports.
Abstract (translated)
医学领域的自动化文本生成是一个活跃的研究和开发领域；然而，评估所生成报告的临床质量仍然是一个挑战，尤其是在缺乏特定领域指标的情况下，例如组织病理学。我们提出了HARE（Histopathology Automated Report Evaluation），这是一种新颖以实体和关系为中心的框架，包括基准数据集、命名实体识别（NER）模型、关系抽取（RE）模型以及一种新的度量标准，该标准通过在参考报告与生成报告之间对关键组织病理学实体和关系进行对齐来优先考虑临床相关内容。为了开发HARE基准，我们为813份去身份化的临床诊断组织病理学报告和来自癌症基因组图谱（TCGA）的652份组织病理学报告进行了特定领域的实体和关系标注。我们将GatorTronS（一种领域适应的语言模型）微调以开发出具有最高整体F1分数（0.915，这是在测试的所有模型中最高的）的HARE-NER和HARE-RE。我们提出的HARE度量标准优于传统的包括ROUGE和Meteor在内的度量标准，以及放射学指标例如RadGraph-XL，在相关性、回归到专家评估方面表现出最高水平（高于第二种最佳方法GREEN，一种大型语言模型驱动的放射报告评估器，通过Pearson $r = 0.168$，Spearman $\rho = 0.161$，Kendall $\tau = 0.123$，$R^2 = 0.176$，$RMSE = 0.018$）。我们已经在[此处链接]发布HARE、数据集和模型，以促进组织病理学报告生成的改进，并为提高报告质量提供了强大的框架。
URL
https://arxiv.org/abs/2509.16326
