Abstract
The scarcity of annotated datasets for clinical information extraction in non-English languages hinders the evaluation of large language model (LLM)-based methods developed primarily in English. In this study, we present the first comprehensive bilingual evaluation of LLMs for the clinical Relation Extraction (RE) task in both English and Turkish. To facilitate this evaluation, we introduce the first English-Turkish parallel clinical RE dataset, derived and carefully curated from the 2010 i2b2/VA relation classification corpus. We systematically assess a diverse set of prompting strategies, including multiple in-context learning (ICL) and Chain-of-Thought (CoT) approaches, and compare their performance to fine-tuned baselines such as PURE. Furthermore, we propose Relation-Aware Retrieval (RAR), a novel in-context example selection method based on contrastive learning, that is specifically designed to capture both sentence-level and relation-level semantics. Our results show that prompting-based LLM approaches consistently outperform traditional fine-tuned models. Moreover, evaluations for English performed better than their Turkish counterparts across all evaluated LLMs and prompting techniques. Among ICL methods, RAR achieves the highest performance, with Gemini 1.5 Flash reaching a micro-F1 score of 0.906 in English and 0.888 in Turkish. Performance further improves to 0.918 F1 in English when RAR is combined with a structured reasoning prompt using the DeepSeek-V3 model. These findings highlight the importance of high-quality demonstration retrieval and underscore the potential of advanced retrieval and prompting techniques to bridge resource gaps in clinical natural language processing.
Abstract (translated)
在非英语语言中，用于临床信息提取的注释数据集的稀缺性阻碍了主要用英语开发的大规模语言模型（LLM）方法的评估。在这项研究中，我们首次提出了针对英语和土耳其语的临床关系抽取（RE）任务的全面双语评价。为了支持这一评估，我们引入了第一个英土双语文本平行的临床RE数据集，该数据集源自2010年i2b2/VA关系分类语料库，并经过仔细整理。我们系统地评估了一系列不同的提示策略，包括多种上下文学习（ICL）和链式思维（CoT）方法，并将其性能与微调基准模型（如PURE）进行了比较。此外，我们提出了基于对比学习的关系感知检索（RAR），这是一种新型的在上下文示例选择方法，特别设计用于捕捉句子级别和关系级别的语义。我们的结果表明，基于提示的LLM方法始终优于传统的微调模型。而且，在所有评估的LLM和提示技术中，英语的性能都优于土耳其语。在ICL方法中，RAR取得了最高的表现，其中Gemini 1.5 Flash在英语中的微平均F1得分为0.906，在土耳其语中为0.888。当RAR与使用DeepSeek-V3模型的结构化推理提示结合时，性能进一步提高至英语中的0.918 F1得分。这些发现突显了高质量演示检索的重要性，并强调了先进的检索和提示技术在临床自然语言处理资源缺口方面具有巨大的潜力。
URL
https://arxiv.org/abs/2601.09367
