Relation_Extraction
-
LLM-OREF: An Open Relation Extraction Framework Based on Large Language Models
2025-09-18 15:46:40Hongyao Tu, Liang Zhang, Yujie Lin, Xin Lin, Haibo Zhang, Long Zhang, Jinsong SuAbstract
The goal of open relation extraction (OpenRE) is to develop an RE model that can generalize to new relations not encountered during training. Existing studies primarily formulate OpenRE as a clustering task. They first cluster all test instances based on the similarity between the instances, and then manually assign a new relation to each cluster. However, their reliance on human annotation limits their practicality. In this paper, we propose an OpenRE framework based on large language models (LLMs), which directly predicts new relations for test instances by leveraging their strong language understanding and generation abilities, without human intervention. Specifically, our framework consists of two core components: (1) a relation discoverer (RD), designed to predict new relations for test instances based on \textit{demonstrations} formed by training instances with known relations; and (2) a relation predictor (RP), used to select the most likely relation for a test instance from $n$ candidate relations, guided by \textit{demonstrations} composed of their instances. To enhance the ability of our framework to predict new relations, we design a self-correcting inference strategy composed of three stages: relation discovery, relation denoising, and relation prediction. In the first stage, we use RD to preliminarily predict new relations for all test instances. Next, we apply RP to select some high-reliability test instances for each new relation from the prediction results of RD through a cross-validation method. During the third stage, we employ RP to re-predict the relations of all test instances based on the demonstrations constructed from these reliable test instances. Extensive experiments on three OpenRE datasets demonstrate the effectiveness of our framework. We release our code at this https URL.
Abstract (translated)
开放关系抽取（Open Relation Extraction，简称OpenRE）的目标是开发出能够泛化到训练过程中未曾遇到的新关系的实体关系模型。现有研究主要将OpenRE视为一个聚类任务，即先根据实例间的相似度对所有测试用例进行聚类，然后人工为每个簇分配一个新的关系。然而，这种依赖于人力标注的方法限制了其实际应用性。 本文提出了一种基于大规模语言模型（Large Language Models, LLMs）的OpenRE框架，该框架能够直接利用LLMs强大的语言理解和生成能力预测测试实例的新关系，无需人工干预。具体而言，我们的框架包含两个核心组件： 1. 关系发现器 (Relation Discoverer, RD)：用于基于训练实例与已知关系形成的“示例”来预测新关系。 2. 关系预测器 (Relation Predictor, RP)：用于从 $n$ 个候选关系中选择最有可能的关系，通过构成这些实例的“示例”进行指导。 为了增强我们框架预测新关系的能力，设计了一种三阶段自校正推理策略，包括：关系发现、去噪和关系预测。在第一阶段，使用RD初步为所有测试用例预测新关系。接下来，在第二阶段中，利用RP通过交叉验证方法从RD的预测结果中挑选出每个新关系的一些高可信度的测试实例。最后，在第三阶段，使用RP基于这些可靠测试用例构建的“示例”重新预测所有测试用例的关系。 在三个OpenRE数据集上的广泛实验展示了我们框架的有效性，并且我们在[这个链接](https://this.http URL/)发布了我们的代码。
URL
https://arxiv.org/abs/2509.15089
PDF
-
Transformer Enhanced Relation Classification: A Comparative Analysis of Contextuality, Data Efficiency and Sequence Complexity
2025-09-14 18:11:31Bowen Jing, Yang Cui, Tianpeng HuangAbstract
In the era of large language model, relation extraction (RE) plays an important role in information extraction through the transformation of unstructured raw text into structured data (Wadhwa et al., 2023). In this paper, we systematically compare the performance of deep supervised learning approaches without transformers and those with transformers. We used a series of non-transformer architectures such as PA-LSTM(Zhang et al., 2017), C-GCN(Zhang et al., 2018), and AGGCN(attention guide GCN)(Guo et al., 2019), and a series of transformer architectures such as BERT, RoBERTa, and R-BERT(Wu and He, 2019). Our comparison included traditional metrics like micro F1, as well as evaluations in different scenarios, varying sentence lengths, and different percentages of the dataset for training. Our experiments were conducted on TACRED, TACREV, and RE-TACRED. The results show that transformer-based models outperform non-transformer models, achieving micro F1 scores of 80-90% compared to 64-67% for non-transformer models. Additionally, we briefly review the research journey in supervised relation classification and discuss the role and current status of large language models (LLMs) in relation extraction.
Abstract (translated)
在大型语言模型时代，关系抽取（RE）通过将非结构化的原始文本转换为结构化数据，在信息提取中扮演着重要角色（Wadhwa等人，2023）。本文系统地比较了没有使用变换器的深度监督学习方法与使用变换器的方法之间的性能。我们采用了一系列非变换器架构，如PA-LSTM(Zhang等人，2017)，C-GCN(Zhang等人，2018)和AGGCN（注意力引导GCN）(Guo等人，2019)，以及一系列基于变换器的架构，例如BERT、RoBERTa和R-BERT(Wu和He, 2019)。我们的比较包括传统的指标如微平均F1分数，以及不同场景下的评估、不同的句子长度和数据集中用于训练的不同百分比的数据集。我们在TACRED、TACREV和RE-TACRED上进行了实验。结果显示，基于变换器的模型优于非变换器模型，在微平均F1分数方面达到了80%-90%，而后者仅为64%-67%。此外，我们还简要回顾了监督关系分类的研究历程，并讨论了大规模语言模型（LLMs）在关系抽取中的作用和当前地位。
URL
https://arxiv.org/abs/2509.11374
PDF
-
CommonVoice-SpeechRE and RPG-MoGe: Advancing Speech Relation Extraction with a New Dataset and Multi-Order Generative Framework
2025-09-10 09:35:43Jinzhong Ning, Paerhati Tulajiang, Yingying Le, Yijia Zhang, Yuanyuan Sun, Hongfei Lin, Haifeng LiuAbstract
Speech Relation Extraction (SpeechRE) aims to extract relation triplets directly from speech. However, existing benchmark datasets rely heavily on synthetic data, lacking sufficient quantity and diversity of real human speech. Moreover, existing models also suffer from rigid single-order generation templates and weak semantic alignment, substantially limiting their performance. To address these challenges, we introduce CommonVoice-SpeechRE, a large-scale dataset comprising nearly 20,000 real-human speech samples from diverse speakers, establishing a new benchmark for SpeechRE research. Furthermore, we propose the Relation Prompt-Guided Multi-Order Generative Ensemble (RPG-MoGe), a novel framework that features: (1) a multi-order triplet generation ensemble strategy, leveraging data diversity through diverse element orders during both training and inference, and (2) CNN-based latent relation prediction heads that generate explicit relation prompts to guide cross-modal alignment and accurate triplet generation. Experiments show our approach outperforms state-of-the-art methods, providing both a benchmark dataset and an effective solution for real-world SpeechRE. The source code and dataset are publicly available at this https URL.
Abstract (translated)
Speech Relation Extraction（SpeechRE）的目标是从语音中直接提取关系三元组。然而，现有的基准数据集严重依赖于合成数据，缺少足够数量和多样性的真人语音样本。此外，现有模型还受到单一生成模板的限制，并且语义对齐能力较弱，这极大地限制了它们的表现。为了应对这些挑战，我们引入了CommonVoice-SpeechRE，这是一个大规模的数据集，包含近20,000个来自多样化发言人的真实人类语音样本，为SpeechRE研究建立了新的基准。 此外，我们还提出了Relation Prompt-Guided Multi-Order Generative Ensemble（RPG-MoGe），这是一种新型框架，具有以下特点：(1) 一个多阶三元组生成集成策略，在训练和推断过程中通过多样化的元素顺序利用数据多样性；(2) 基于CNN的潜在关系预测头，这些头部可以生成明确的关系提示以指导跨模态对齐，并准确生成三元组。实验表明我们的方法超越了当前最先进的方法，不仅提供了一个基准数据集，还为现实世界的SpeechRE问题提供了有效的解决方案。 源代码和数据集在以下网址公开：[此处应填写具体链接]。
URL
https://arxiv.org/abs/2509.08438
PDF
-
SciNLP: A Domain-Specific Benchmark for Full-Text Scientific Entity and Relation Extraction in NLP
2025-09-09 14:41:40Decheng Duan, Yingyi Zhang, Jitong Peng, Chengzhi ZhangAbstract
Structured information extraction from scientific literature is crucial for capturing core concepts and emerging trends in specialized fields. While existing datasets aid model development, most focus on specific publication sections due to domain complexity and the high cost of annotating scientific texts. To address this limitation, we introduce SciNLP - a specialized benchmark for full-text entity and relation extraction in the Natural Language Processing (NLP) domain. The dataset comprises 60 manually annotated full-text NLP publications, covering 7,072 entities and 1,826 relations. Compared to existing research, SciNLP is the first dataset providing full-text annotations of entities and their relationships in the NLP domain. To validate the effectiveness of SciNLP, we conducted comparative experiments with similar datasets and evaluated the performance of state-of-the-art supervised models on this dataset. Results reveal varying extraction capabilities of existing models across academic texts of different lengths. Cross-comparisons with existing datasets show that SciNLP achieves significant performance improvements on certain baseline models. Using models trained on SciNLP, we implemented automatic construction of a fine-grained knowledge graph for the NLP domain. Our KG has an average node degree of 3.2 per entity, indicating rich semantic topological information that enhances downstream applications. The dataset is publicly available at this https URL.
Abstract (translated)
从科学文献中提取结构化信息对于捕捉特定领域的核心概念和新兴趋势至关重要。尽管现有的数据集有助于模型开发，但大多数现有数据集由于领域复杂性和标注科学文本的高昂成本而仅侧重于具体出版物的部分内容。为了克服这一限制，我们引入了SciNLP——这是一个专门用于自然语言处理（NLP）领域全文实体与关系提取的专业基准测试。该数据集包括60篇手写的、完整注释的NLP出版物，覆盖7,072个实体和1,826种关系。相比于现有研究，SciNLP是首个提供NLP领域内全文本中的实体及其关系标注的数据集。 为了验证SciNLP的有效性，我们进行了与类似数据集的比较实验，并评估了最先进的监督模型在该数据集上的性能表现。结果表明，现有的模型对于不同长度学术文本的信息抽取能力存在差异。交叉对比现有数据集显示，SciNLP对某些基准模型实现了显著的性能提升。 利用基于SciNLP训练的模型，我们自动构建了一个针对NLP领域的细粒度知识图谱（KG）。我们的KG每个实体平均有3.2个关联节点，显示出丰富的语义拓扑信息，这有助于下游应用的发展。该数据集在以下链接公开可得：[提供URL]
URL
https://arxiv.org/abs/2509.07801
PDF
-
M-BRe: Discovering Training Samples for Relation Extraction from Unlabeled Texts with Large Language Models
2025-09-09 13:32:29Zexuan Li, Hongliang Dai, Piji LiAbstract
For Relation Extraction (RE), the manual annotation of training data may be prohibitively expensive, since the sentences that contain the target relations in texts can be very scarce and difficult to find. It is therefore beneficial to develop an efficient method that can automatically extract training instances from unlabeled texts for training RE models. Recently, large language models (LLMs) have been adopted in various natural language processing tasks, with RE also benefiting from their advances. However, when leveraging LLMs for RE with predefined relation categories, two key challenges arise. First, in a multi-class classification setting, LLMs often struggle to comprehensively capture the semantics of every relation, leading to suboptimal results. Second, although employing binary classification for each relation individually can mitigate this issue, it introduces significant computational overhead, resulting in impractical time complexity for real-world applications. Therefore, this paper proposes a framework called M-BRe to extract training instances from unlabeled texts for RE. It utilizes three modules to combine the advantages of both of the above classification approaches: Relation Grouping, Relation Extraction, and Label Decision. Extensive experiments confirm its superior capability in discovering high-quality training samples from unlabeled texts for RE.
Abstract (translated)
对于关系抽取（RE），训练数据的手动标注可能成本极高，因为文本中包含目标关系的句子可能会非常稀少且难以找到。因此，开发一种可以从未标记文本中自动提取训练实例的方法以用于训练RE模型是有益的。最近，大型语言模型（LLMs）已被应用于各种自然语言处理任务，并且RE也从中受益匪浅。然而，在利用LLMs进行关系抽取时，特别是在使用预定义的关系类别的情况下，会出现两个关键挑战。首先，在多类分类设置中，LLMs往往难以全面捕捉每种关系的语义，从而导致次优结果。其次，尽管对每个关系单独采用二元分类可以缓解这一问题，但这种方法会引入显著的计算开销，并导致实际应用中的时间复杂度不可行。因此，本文提出了一种名为M-BRe的框架，用于从未标记文本中提取训练实例以进行RE。该框架利用三个模块：关系分组、关系抽取和标签决策，结合上述两种分类方法的优点。广泛的实验验证了其在从未标记文本中发现高质量训练样本方面的卓越能力。
URL
https://arxiv.org/abs/2509.07730
PDF
-
REMOTE: A Unified Multimodal Relation Extraction Framework with Multilevel Optimal Transport and Mixture-of-Experts
2025-09-05 06:52:03Xinkui Lin, Yongxiu Xu, Minghao Tang, Shilong Zhang, Hongbo Xu, Hao Xu, Yubin WangAbstract
Multimodal relation extraction (MRE) is a crucial task in the fields of Knowledge Graph and Multimedia, playing a pivotal role in multimodal knowledge graph construction. However, existing methods are typically limited to extracting a single type of relational triplet, which restricts their ability to extract triplets beyond the specified types. Directly combining these methods fails to capture dynamic cross-modal interactions and introduces significant computational redundancy. Therefore, we propose a novel \textit{unified multimodal Relation Extraction framework with Multilevel Optimal Transport and mixture-of-Experts}, termed REMOTE, which can simultaneously extract intra-modal and inter-modal relations between textual entities and visual objects. To dynamically select optimal interaction features for different types of relational triplets, we introduce mixture-of-experts mechanism, ensuring the most relevant modality information is utilized. Additionally, considering that the inherent property of multilayer sequential encoding in existing encoders often leads to the loss of low-level information, we adopt a multilevel optimal transport fusion module to preserve low-level features while maintaining multilayer encoding, yielding more expressive representations. Correspondingly, we also create a Unified Multimodal Relation Extraction (UMRE) dataset to evaluate the effectiveness of our framework, encompassing diverse cases where the head and tail entities can originate from either text or image. Extensive experiments show that REMOTE effectively extracts various types of relational triplets and achieves state-of-the-art performanc on almost all metrics across two other public MRE datasets. We release our resources at this https URL.
Abstract (translated)
多模态关系抽取（MRE）是知识图谱和多媒体领域中的一个关键任务，在多模态知识图构建中扮演着重要角色。然而，现有的方法通常只能提取单一类型的三元组关系，这限制了它们在指定类型之外的三元组提取能力。直接组合这些方法无法捕捉动态跨模式交互，并且会引入显著的计算冗余。因此，我们提出了一种新型框架——“多级最优传输与专家混合型统一多模态关系抽取”（REMOTE），它可以同时提取文本实体和视觉对象之间的同模态和异模态关系。 为了为不同类型的三元组动态选择最佳交互特征，我们在混合专家机制中引入了这一方法，确保最相关的模式信息被利用。此外，鉴于现有编码器的多层序列编码特性往往会失去低级信息，我们采用了多层次最优传输融合模块来保留低级特征的同时保持多层编码，从而产生更具有表现力的表示形式。 相应地，为了评估我们的框架的有效性，我们还创建了一个统一的多模态关系抽取（UMRE）数据集，涵盖了各种情况，在这些情况下，头部和尾部实体可以分别来自文本或图像。广泛的实验表明，REMOTE能够有效地提取各种类型的三元组，并在其他两个公开MRE数据集中几乎所有度量标准上均实现了最先进的性能。 我们已将资源发布在此链接：[this https URL]（请根据实际需求替换具体的URL地址）。
URL
https://arxiv.org/abs/2509.04844
PDF
-
A Study of Large Language Models for Patient Information Extraction: Model Architecture, Fine-Tuning Strategy, and Multi-task Instruction Tuning
2025-09-05 02:07:40Cheng Peng, Xinyu Dong, Mengxian Lyu, Daniel Paredes, Yaoyun Zhang, Yonghui WuAbstract
Natural language processing (NLP) is a key technology to extract important patient information from clinical narratives to support healthcare applications. The rapid development of large language models (LLMs) has revolutionized many NLP tasks in the clinical domain, yet their optimal use in patient information extraction tasks requires further exploration. This study examines LLMs' effectiveness in patient information extraction, focusing on LLM architectures, fine-tuning strategies, and multi-task instruction tuning techniques for developing robust and generalizable patient information extraction systems. This study aims to explore key concepts of using LLMs for clinical concept and relation extraction tasks, including: (1) encoder-only or decoder-only LLMs, (2) prompt-based parameter-efficient fine-tuning (PEFT) algorithms, and (3) multi-task instruction tuning on few-shot learning performance. We benchmarked a suite of LLMs, including encoder-based LLMs (BERT, GatorTron) and decoder-based LLMs (GatorTronGPT, Llama 3.1, GatorTronLlama), across five datasets. We compared traditional full-size fine-tuning and prompt-based PEFT. We explored a multi-task instruction tuning framework that combines both tasks across four datasets to evaluate the zero-shot and few-shot learning performance using the leave-one-dataset-out strategy.
Abstract (translated)
自然语言处理（NLP）是提取临床叙述中重要患者信息以支持医疗应用的关键技术。大型语言模型（LLMs）的快速发展已经革新了临床领域中的许多NLP任务，但它们在患者信息提取任务中最优使用方法仍需进一步探索。本研究旨在考察LLMs在患者信息提取方面的有效性，重点关注LLM架构、微调策略以及多任务指令调整技术，以开发稳健且通用的患者信息抽取系统。该研究探讨了使用LLMs进行临床概念和关系提取任务的关键概念，包括：(1)仅编码器或仅解码器LLMs；(2)基于提示的参数高效微调（PEFT）算法；以及(3)在少量样本学习性能上的多任务指令调整技术。我们对一系列LLM进行了基准测试，其中包括基于编码器的模型（BERT、GatorTron）和基于解码器的模型（GatorTronGPT、Llama 3.1、GatorTronLlama），这些模型均在五个数据集上进行评估。我们对比了传统的全尺寸微调与基于提示的PEFT方法，并探索了一种多任务指令调整框架，该框架结合四个数据集中跨任务的任务，使用leave-one-dataset-out策略来评估零样本和少量样本学习性能。
URL
https://arxiv.org/abs/2509.04753
PDF
-
Quantized Large Language Models in Biomedical Natural Language Processing: Evaluation and Recommendation
2025-09-04 04:18:45Zaifu Zhan, Shuang Zhou, Min Zeng, Kai Yu, Meijia Song, Xiaoyi Chen, Jun Wang, Yu Hou, Rui ZhangAbstract
Large language models have demonstrated remarkable capabilities in biomedical natural language processing, yet their rapid growth in size and computational requirements present a major barrier to adoption in healthcare settings where data privacy precludes cloud deployment and resources are limited. In this study, we systematically evaluated the impact of quantization on 12 state-of-the-art large language models, including both general-purpose and biomedical-specific models, across eight benchmark datasets covering four key tasks: named entity recognition, relation extraction, multi-label classification, and question answering. We show that quantization substantially reduces GPU memory requirements-by up to 75%-while preserving model performance across diverse tasks, enabling the deployment of 70B-parameter models on 40GB consumer-grade GPUs. In addition, domain-specific knowledge and responsiveness to advanced prompting methods are largely maintained. These findings provide significant practical and guiding value, highlighting quantization as a practical and effective strategy for enabling the secure, local deployment of large yet high-capacity language models in biomedical contexts, bridging the gap between technical advances in AI and real-world clinical translation.
Abstract (translated)
大型语言模型在生物医学自然语言处理方面展示了非凡的能力，但其规模迅速扩大和计算需求增加，在数据隐私限制云部署且资源有限的医疗环境中构成了主要障碍。在这项研究中，我们系统地评估了量化对12种最先进的大规模语言模型（包括通用型和生物医学专用型）的影响，这些模型在涵盖四个关键任务的八个基准数据集上进行了测试：命名实体识别、关系抽取、多标签分类和问答。研究表明，量化技术能够显著减少GPU内存需求（最高可达75%），同时保持模型在多种任务上的性能，使得拥有700亿参数的大规模模型能够在40GB消费级GPU上运行。此外，领域特定知识以及对先进提示方法的响应性也得到了很大程度的保留。 这些发现提供了重要的实用和指导价值，突显了量化作为使大型且高性能语言模型在生物医学环境中实现安全本地部署的一种实际有效策略的重要性，从而弥合了人工智能技术进步与现实世界临床应用之间的差距。
URL
https://arxiv.org/abs/2509.04534
PDF
-
Continuous Monitoring of Large-Scale Generative AI via Deterministic Knowledge Graph Structures
2025-09-04 03:34:49Kishor Datta Gupta, Mohd Ariful Haque, Hasmot Ali, Marufa Kamal, Syed Bahauddin Alam, Mohammad Ashiqur RahmanAbstract
Generative AI (GEN AI) models have revolutionized diverse application domains but present substantial challenges due to reliability concerns, including hallucinations, semantic drift, and inherent biases. These models typically operate as black-boxes, complicating transparent and objective evaluation. Current evaluation methods primarily depend on subjective human assessment, limiting scalability, transparency, and effectiveness. This research proposes a systematic methodology using deterministic and Large Language Model (LLM)-generated Knowledge Graphs (KGs) to continuously monitor and evaluate GEN AI reliability. We construct two parallel KGs: (i) a deterministic KG built using explicit rule-based methods, predefined ontologies, domain-specific dictionaries, and structured entity-relation extraction rules, and (ii) an LLM-generated KG dynamically derived from real-time textual data streams such as live news articles. Utilizing real-time news streams ensures authenticity, mitigates biases from repetitive training, and prevents adaptive LLMs from bypassing predefined benchmarks through feedback memorization. To quantify structural deviations and semantic discrepancies, we employ several established KG metrics, including Instantiated Class Ratio (ICR), Instantiated Property Ratio (IPR), and Class Instantiation (CI). An automated real-time monitoring framework continuously computes deviations between deterministic and LLM-generated KGs. By establishing dynamic anomaly thresholds based on historical structural metric distributions, our method proactively identifies and flags significant deviations, thus promptly detecting semantic anomalies or hallucinations. This structured, metric-driven comparison between deterministic and dynamically generated KGs delivers a robust and scalable evaluation framework.
Abstract (translated)
生成式人工智能（GEN AI）模型已在多个应用领域实现了革命性的变革，但它们由于可靠性问题而面临重大挑战，这些问题包括幻觉、语义漂移和固有偏见。这些模型通常以黑箱形式运作，这使得透明且客观的评估变得复杂化。当前的评估方法主要依赖于主观的人类评判，从而限制了其可扩展性、透明度和有效性。本研究提出了一种系统的方法，利用确定性的知识图谱（KG）和大型语言模型（LLM）生成的知识图谱来持续监控和评估GEN AI的可靠性。我们构建了两个平行的知识图谱：(i) 一个使用明确规则基于方法、预定义的本体论、领域特定字典以及结构化的实体-关系提取规则构建的确定性知识图谱，(ii) 一个从实时文本流（如即时新闻文章）中动态生成的LLM知识图谱。利用实时新闻流确保了真实性，并且可以缓解重复训练导致的偏见问题，防止通过反馈记忆使适应性强的LLM绕过预定义基准的问题。 为了量化结构偏差和语义差异，我们采用了几种已建立的知识图谱指标，包括实例化类比例（ICR）、实例化属性比例（IPR）以及类实例化（CI）。一个自动化的实时监控框架持续计算确定性知识图谱与LLM生成的知识图谱之间的偏差。通过基于历史结构指标分布来动态设置异常阈值，我们的方法能够提前识别和标记重要的差异，并及时检测语义异常或幻觉。 这种基于结构化、度量驱动的方法对确定性和动态生成的知识图谱进行比较，为评估提供了强大且可扩展的框架。
URL
https://arxiv.org/abs/2509.03857
PDF
-
The Temporal Game: A New Perspective on Temporal Relation Extraction
2025-08-29 21:33:07Hugo Sousa, Ricardo Campos, Al\'ipio JorgeAbstract
In this paper we demo the Temporal Game, a novel approach to temporal relation extraction that casts the task as an interactive game. Instead of directly annotating interval-level relations, our approach decomposes them into point-wise comparisons between the start and end points of temporal entities. At each step, players classify a single point relation, and the system applies temporal closure to infer additional relations and enforce consistency. This point-based strategy naturally supports both interval and instant entities, enabling more fine-grained and flexible annotation than any previous approach. The Temporal Game also lays the groundwork for training reinforcement learning agents, by treating temporal annotation as a sequential decision-making task. To showcase this potential, the demo presented in this paper includes a Game mode, in which users annotate texts from the TempEval-3 dataset and receive feedback based on a scoring system, and an Annotation mode, that allows custom documents to be annotated and resulting timeline to be exported. Therefore, this demo serves both as a research tool and an annotation interface. The demo is publicly available at this https URL, and the source code is open-sourced to foster further research and community-driven development in temporal reasoning and annotation.
Abstract (translated)
在这篇论文中，我们展示了Temporal Game，这是一种新颖的时序关系抽取方法，它将任务视为互动游戏。我们的方法不是直接标注区间级别的关系，而是将其分解为时间实体起始点和终点之间的单点比较。在每一步中，玩家都会对单一的点关系进行分类，系统则通过应用时序闭包来推断额外的关系并确保一致性。这种基于点的方法自然支持间隔和瞬间实体，提供了比以往任何方法都更精细、更灵活的标注功能。Temporal Game也为训练强化学习代理奠定了基础，它将时间标注视为一个顺序决策任务。为了展示这一潜力，本文所呈现的演示包括一个Game模式，在此模式中用户可以对TempEval-3数据集中的文本进行注释，并根据评分系统接收反馈；以及一个Annotation模式，允许自定义文档被标注并且生成的时间线可导出。因此，该演示既是一种研究工具，也是一项注释界面。 公开版的Temporal Game可以在[此处](https://this https URL/)访问，源代码也是开源的，旨在促进时序推理和社区驱动的发展方面的进一步研究。
URL
https://arxiv.org/abs/2509.00250
PDF
-
Re-Representation in Sentential Relation Extraction with Sequence Routing Algorithm
2025-08-28 17:54:35Ramazan Ali Bahrami, Ramin YahyapourAbstract
Sentential relation extraction (RE) is an important task in natural language processing (NLP). In this paper we propose to do sentential RE with dynamic routing in capsules. We first show that the proposed approach outperform state of the art on common sentential relation extraction datasets Tacred, Tacredrev, Retacred, and Conll04. We then investigate potential reasons for its good performance on the mentioned datasets, and yet low performance on another similar, yet larger sentential RE dataset, Wikidata. As such, we identify noise in Wikidata labels as one of the reasons that can hinder performance. Additionally, we show associativity of better performance with better re-representation, a term from neuroscience referred to change of representation in human brain to improve the match at comparison time. As example, in the given analogous terms King:Queen::Man:Woman, at comparison time, and as a result of re-representation, the similarity between related head terms (King,Man), and tail terms (Queen,Woman) increases. As such, our observation show that our proposed model can do re-representation better than the vanilla model compared with. To that end, beside noise in the labels of the distantly supervised RE datasets, we propose re-representation as a challenge in sentential RE.
Abstract (translated)
句子关系抽取（RE）是自然语言处理（NLP）中的一个重要任务。在本文中，我们提出了一种使用胶囊动态路由进行句子级关系抽取的方法。首先，我们展示了所提出的这种方法在常见句子级关系抽取数据集Tacred、Tacredrev、Retacred和Conll04上优于当前最佳性能。然后，我们探讨了它在这类数据集中表现优异的原因，同时在另一个规模更大但相似的句子级RE数据集Wikidata上却表现出较差性能的原因。通过这样的分析，我们识别出Wikidata标签中的噪声可能是影响其性能的一个重要原因。 此外，我们还发现更好的重新表示与更优的表现有相关性，这是神经科学中用来指代人类大脑在比较时改变表示以提高匹配度的概念。“重新表示”一词意指：例如，在给定的类比项King:Queen::Man:Woman（国王：王后 :: 男人：女人）中，通过重新表示，头部术语（King和Man）之间以及尾部术语（Queen和Woman）之间的相似性增加。因此，我们的观察表明，与传统模型相比，我们提出的方法能够更好地进行重新表示。 最后，在除了距离监督RE数据集中标签的噪声之外，我们认为重新表示是句子级关系抽取中的一个挑战。
URL
https://arxiv.org/abs/2508.21049
PDF
-
GDLLM: A Global Distance-aware Modeling Approach Based on Large Language Models for Event Temporal Relation Extraction
2025-08-28 14:23:39Jie Zhao, Wanting Ning, Yuxiao Fei, Yubo Feng, Lishuang LiAbstract
In Natural Language Processing(NLP), Event Temporal Relation Extraction (ETRE) is to recognize the temporal relations of two events. Prior studies have noted the importance of language models for ETRE. However, the restricted pre-trained knowledge of Small Language Models(SLMs) limits their capability to handle minority class relations in imbalanced classification datasets. For Large Language Models(LLMs), researchers adopt manually designed prompts or instructions, which may introduce extra noise, leading to interference with the model's judgment of the long-distance dependencies between events. To address these issues, we propose GDLLM, a Global Distance-aware modeling approach based on LLMs. We first present a distance-aware graph structure utilizing Graph Attention Network(GAT) to assist the LLMs in capturing long-distance dependency features. Additionally, we design a temporal feature learning paradigm based on soft inference to augment the identification of relations with a short-distance proximity band, which supplements the probabilistic information generated by LLMs into the multi-head attention mechanism. Since the global feature can be captured effectively, our framework substantially enhances the performance of minority relation classes and improves the overall learning ability. Experiments on two publicly available datasets, TB-Dense and MATRES, demonstrate that our approach achieves state-of-the-art (SOTA) performance.
Abstract (translated)
在自然语言处理（NLP）领域，事件时间关系抽取（ETRE）旨在识别两个事件之间的时间关系。先前的研究已经注意到了语言模型对于ETRE的重要性。然而，小型语言模型（SLM）有限的预训练知识限制了它们处理不平衡分类数据集中少数类别的能力。对于大型语言模型（LLM），研究人员采用手动设计的提示或指令，这可能会引入额外的噪声，从而干扰模型对事件之间长距离依赖关系的判断。 为了解决这些问题，我们提出了基于LLM的GDLLM方法，这是一种全局距离感知建模方法。首先，我们提出了一种利用图注意力网络（GAT）的距离感知图形结构来帮助LLM捕捉长时间依赖特征。此外，我们设计了一个基于软推理的时间特性学习范式，以增强对具有短距离接近度带的关系的识别，并将由LLMs生成的概率信息补充到多头注意机制中。由于可以有效地捕获全局特征，我们的框架显著提升了少数关系类别的性能并改进了整体学习能力。 在两个公开数据集TB-Dense和MATRES上的实验表明，我们提出的方法达到了最先进的（SOTA）性能。
URL
https://arxiv.org/abs/2508.20828
PDF
-
Benchmarking GPT-5 for biomedical natural language processing
2025-08-28 13:06:53Yu Hou, Zaifu Zhan, Rui ZhangAbstract
The rapid expansion of biomedical literature has heightened the need for scalable natural language processing (NLP) solutions. While GPT-4 substantially narrowed the gap with task-specific systems, especially in question answering, its performance across other domains remained uneven. We updated a standardized BioNLP benchmark to evaluate GPT-5 and GPT-4o under zero-, one-, and five-shot prompting across 12 datasets spanning six task families: named entity recognition, relation extraction, multi-label document classification, question answering, text summarization, and text simplification. Using fixed prompt templates, identical decoding parameters, and batch inference, we report primary metrics per dataset and include prior results for GPT-4, GPT-3.5, and LLaMA-2-13B for comparison. GPT-5 achieved the strongest overall benchmark performance, with macro-average scores rising to 0.557 under five-shot prompting versus 0.506 for GPT-4 and 0.508 for GPT-4o. On MedQA, GPT-5 reached 94.1% accuracy, exceeding the previous supervised state of the art by over fifty points, and attained parity with supervised systems on PubMedQA (0.734). In extraction tasks, GPT-5 delivered major gains in chemical NER (0.886 F1) and ChemProt relation extraction (0.616 F1), outperforming GPT-4 and GPT-4o, though summarization and disease NER still lagged behind domain-specific baselines. These results establish GPT-5 as a general-purpose model now offering deployment-ready performance for reasoning-oriented biomedical QA, while precision-critical extraction and evidence-dense summarization continue to favor fine-tuned or hybrid approaches. The benchmark delineates where simple prompting suffices and where retrieval-augmented or planning-based scaffolds are likely required, providing actionable guidance for BioNLP system design as frontier models advance.
Abstract (translated)
生物医学文献的快速扩张增强了对可扩展自然语言处理（NLP）解决方案的需求。虽然GPT-4在特定任务系统方面，特别是在问题回答上大大缩小了差距，但在其他领域的表现仍然参差不齐。我们更新了一个标准化的BioNLP基准测试，以评估GPT-5和GPT-4o在零样本、一拍和五拍提示下跨12个数据集的表现，这些数据集涵盖了六个任务家族：命名实体识别、关系抽取、多标签文档分类、问题回答、文本摘要和文本简化。使用固定的提示模板、相同的解码参数以及批量推理，我们报告了每个数据集的主要指标，并包含了GPT-4、GPT-3.5和LLaMA-2-13B的先前结果进行比较。 在五拍提示下，GPT-5取得了最强的整体基准性能，宏平均分达到了0.557，相比之下，GPT-4为0.506，而GPT-4o为0.508。在MedQA上，GPT-5的准确率达到了94.1%，超过了之前的监督学习最优状态超过五十个百分点，并且在PubMedQA（0.734）中与监督系统持平。在抽取任务中，GPT-5在化学实体识别（NER）方面取得了显著提升（F1为0.886），以及ChemProt关系提取（F1为0.616），超过了GPT-4和GPT-4o的表现；尽管摘要生成和疾病实体识别仍落后于领域特定基准。这些结果表明，GPT-5已成为一种通用模型，现在可以提供适用于以推理为导向的生物医学问答部署就绪性能，而精度关键的提取以及证据密集型总结任务仍然更倾向于微调或混合方法。 该基准测试明确了简单的提示在哪些情况下足够使用，在哪些情况下可能需要检索增强或基于计划的框架，并为BioNLP系统设计提供了可操作性的指导，随着前沿模型的进步。
URL
https://arxiv.org/abs/2509.04462
PDF
-
Bridging Language Gaps: Enhancing Few-Shot Language Adaptation
2025-08-26 22:49:17Philipp Borchert, Jochen De Weerdt, Marie-Francine MoensAbstract
The disparity in language resources poses a challenge in multilingual NLP, with high-resource languages benefiting from extensive data, while low-resource languages lack sufficient data for effective training. Our Contrastive Language Alignment with Prompting (CoLAP) method addresses this gap by integrating contrastive learning with cross-lingual representations, facilitating task-specific knowledge transfer from high-resource to lower-resource languages. The primary advantage of our approach is its data efficiency, enabling rapid adaptation to new languages and reducing the need for large labeled datasets. We conduct experiments with multilingual encoder-only and decoder-only language models on natural language understanding tasks, including natural language inference and relation extraction, evaluating performance across both high- and low-resource languages. Our results demonstrate that CoLAP outperforms few-shot cross-lingual transfer baselines and in-context learning, even with limited available data. This effectively narrows the cross-lingual performance gap, contributing to the development of more efficient multilingual NLP techniques.
Abstract (translated)
语言资源的不均衡是多语种自然语言处理（NLP）面临的一大挑战，高资源语言得益于丰富的数据支持，而低资源语言则因缺乏足够有效的训练数据而处于不利地位。我们提出了一种对比语言对齐与提示方法（Contrastive Language Alignment with Prompting, CoLAP），通过将对比学习与跨语言表示相结合，实现了从高资源语言向低资源语言的特定任务知识转移。我们的方法的主要优势在于其高效的数据利用性，能够快速适应新语言，并且减少了大型标注数据集的需求。 我们在多语种编码器和解码器独立的语言模型上进行了实验，涉及自然语言理解任务（如自然语言推理和关系抽取），并且在高资源与低资源语言的性能上都进行了评估。我们的结果表明，即使是在有限的数据可用性条件下，CoLAP也优于少量样本跨语言迁移基准方法以及上下文学习方法，在一定程度上缩小了跨语种表现差距，并为更有效的多语种NLP技术的发展做出了贡献。
URL
https://arxiv.org/abs/2508.19464
PDF
-
DEPTH: Hallucination-Free Relation Extraction via Dependency-Aware Sentence Simplification and Two-tiered Hierarchical Refinement
2025-08-20 03:35:24Yupei Yang, Fan Feng, Lin Yang, Wanxi Deng, Lin Qu, Biwei Huang, Shikui Tu, Lei XuAbstract
Relation extraction enables the construction of structured knowledge for many downstream applications. While large language models (LLMs) have shown great promise in this domain, most existing methods concentrate on relation classification, which predicts the semantic relation type between a related entity pair. However, we observe that LLMs often struggle to reliably determine whether a relation exists, especially in cases involving complex sentence structures or intricate semantics, which leads to spurious predictions. Such hallucinations can introduce noisy edges in knowledge graphs, compromising the integrity of structured knowledge and downstream reliability. To address these challenges, we propose DEPTH, a framework that integrates Dependency-aware sEntence simPlification and Two-tiered Hierarchical refinement into the relation extraction pipeline. Given a sentence and its candidate entity pairs, DEPTH operates in two stages: (1) the Grounding module extracts relations for each pair by leveraging their shortest dependency path, distilling the sentence into a minimal yet coherent relational context that reduces syntactic noise while preserving key semantics; (2) the Refinement module aggregates all local predictions and revises them based on a holistic understanding of the sentence, correcting omissions and inconsistencies. We further introduce a causality-driven reward model that mitigates reward hacking by disentangling spurious correlations, enabling robust fine-tuning via reinforcement learning with human feedback. Experiments on six benchmarks demonstrate that DEPTH reduces the average hallucination rate to 7.0\% while achieving a 17.2\% improvement in average F1 score over state-of-the-art baselines.
Abstract (translated)
关系抽取使得为许多下游应用构建结构化知识成为可能。尽管大型语言模型（LLM）在该领域展现出巨大潜力，但大多数现有方法主要集中在关系分类上，即预测相关实体对之间的语义关系类型。然而，我们观察到，在涉及复杂句式或细微语义的情况下，LLMs往往难以可靠地确定是否存在某种关系，这会导致错误预测。这些幻觉会在知识图谱中引入噪音边，从而破坏结构化知识的完整性以及下游应用的可靠性。 为了解决这些问题，我们提出了DEPTH框架，该框架将依赖感知句子简化和两级层次细化整合到关系抽取流程中。给定一个句子及其候选实体对，DEPTH分两步操作：（1）定位模块通过利用最短依存路径来提取每个实体对的关系，并将句子精简为一个最小而连贯的关联语境，从而减少句法噪音同时保留关键语义；（2）优化模块汇总所有局部预测结果并基于整个句子的整体理解进行修订，纠正遗漏和不一致之处。我们进一步提出了一种因果驱动的奖励模型，通过分解虚假相关性来防止奖励黑客行为，并支持通过人类反馈强化学习实现稳健微调。 在六个基准上的实验表明，DEPTH将幻觉率平均降低到7.0%，同时相对于最先进的基线方法实现了17.2%的F1分数提升。
URL
https://arxiv.org/abs/2508.14391
PDF
-
Extracting Structured Requirements from Unstructured Building Technical Specifications for Building Information Modeling
2025-08-19 13:55:41Insaf Nahri, Romain Pinqui\'e, Philippe V\'eron, Nicolas Bus, Mathieu ThorelAbstract
This study explores the integration of Building Information Modeling (BIM) with Natural Language Processing (NLP) to automate the extraction of requirements from unstructured French Building Technical Specification (BTS) documents within the construction industry. Employing Named Entity Recognition (NER) and Relation Extraction (RE) techniques, the study leverages the transformer-based model CamemBERT and applies transfer learning with the French language model Fr\_core\_news\_lg, both pre-trained on a large French corpus in the general domain. To benchmark these models, additional approaches ranging from rule-based to deep learning-based methods are developed. For RE, four different supervised models, including Random Forest, are implemented using a custom feature vector. A hand-crafted annotated dataset is used to compare the effectiveness of NER approaches and RE models. Results indicate that CamemBERT and Fr\_core\_news\_lg exhibited superior performance in NER, achieving F1-scores over 90\%, while Random Forest proved most effective in RE, with an F1 score above 80\%. The outcomes are intended to be represented as a knowledge graph in future work to further enhance automatic verification systems.
Abstract (translated)
这项研究探讨了将建筑信息模型（BIM）与自然语言处理（NLP）相结合，以自动化提取法国建筑工程技术规范（BTS）文档中的需求。这些文档是未结构化的，并且是在建筑业中使用的。该研究利用命名实体识别（NER）和关系抽取（RE）的技术手段，采用基于变换器的CamemBERT模型，并使用法语语言模型Fr_core_news_lg进行迁移学习，后者在广泛的通用领域大型语料库上进行了预训练。为了对这些模型进行基准测试，开发了从规则基础方法到深度学习方法的一系列额外方法。 对于关系抽取（RE），采用了一个自定义特征向量，实现了包括随机森林在内的四种不同的监督式模型。使用手工注释的数据集来比较命名实体识别（NER）的方法和关系抽取（RE）的模型的有效性。结果表明，在命名实体识别方面，CamemBERT和Fr_core_news_lg表现出色，F1分数超过90%；而在关系抽取方面，随机森林方法最为有效，其F1分数超过了80%。 研究结果旨在未来的进一步工作中以知识图谱的形式表示出来，从而增强自动验证系统。
URL
https://arxiv.org/abs/2508.13833
PDF
-
An LLM + ASP Workflow for Joint Entity-Relation Extraction
2025-08-18 04:15:35Trang Tran, Trung Hoang Le, Huiping Cao, Tran Cao SonAbstract
Joint entity-relation extraction (JERE) identifies both entities and their relationships simultaneously. Traditional machine-learning based approaches to performing this task require a large corpus of annotated data and lack the ability to easily incorporate domain specific information in the construction of the model. Therefore, creating a model for JERE is often labor intensive, time consuming, and elaboration intolerant. In this paper, we propose harnessing the capabilities of generative pretrained large language models (LLMs) and the knowledge representation and reasoning capabilities of Answer Set Programming (ASP) to perform JERE. We present a generic workflow for JERE using LLMs and ASP. The workflow is generic in the sense that it can be applied for JERE in any domain. It takes advantage of LLM's capability in natural language understanding in that it works directly with unannotated text. It exploits the elaboration tolerant feature of ASP in that no modification of its core program is required when additional domain specific knowledge, in the form of type specifications, is found and needs to be used. We demonstrate the usefulness of the proposed workflow through experiments with limited training data on three well-known benchmarks for JERE. The results of our experiments show that the LLM + ASP workflow is better than state-of-the-art JERE systems in several categories with only 10\% of training data. It is able to achieve a 2.5 times (35\% over 15\%) improvement in the Relation Extraction task for the SciERC corpus, one of the most difficult benchmarks.
Abstract (translated)
联合实体关系抽取（Joint Entity-Relation Extraction，JERE）同时识别实体及其之间的关系。传统基于机器学习的方法执行此任务时需要大量标注数据，并且缺乏轻松整合特定领域信息的能力。因此，创建一个用于JERE的模型通常是劳动密集型、耗时长并且对详细说明不耐受的。在本文中，我们提出利用生成式预训练大型语言模型（LLMs）和答案集编程（ASP）的知识表示与推理能力来执行JERE任务。我们提出了使用LLM和ASP进行JERE的一个通用工作流程。该工作流程是通用性的，在任何领域都可以应用。它利用了LLM在自然语言理解中的优势，可以直接处理未标注的文本；同时，通过其答案集编程（ASP）中的详尽性容忍特性，无需修改核心程序即可轻松添加特定领域的类型规范等知识。 我们通过在三个著名的JERE基准测试中使用有限训练数据进行实验来展示该工作流程的有效性。我们的实验证明，在只有10%的训练数据下，LLM + ASP的工作流优于当前最先进的JERE系统。对于最困难的基准之一——SciERC语料库中的关系抽取任务，它可以实现2.5倍（从15%到35%）的性能提升。
URL
https://arxiv.org/abs/2508.12611
PDF
-
Learning Wisdom from Errors: Promoting LLM's Continual Relation Learning through Exploiting Error Cases
2025-08-16 12:49:11Shaozhe Yin, Jinyu Guo, Kai Shuang, Xia Liu, Ruize OuAbstract
Continual Relation Extraction (CRE) aims to continually learn new emerging relations while avoiding catastrophic forgetting. Existing CRE methods mainly use memory replay and contrastive learning to mitigate catastrophic forgetting. However, these methods do not attach importance to the error cases that can reveal the model's cognitive biases more effectively. To address this issue, we propose an instruction-based continual contrastive tuning approach for Large Language Models (LLMs) in CRE. Different from existing CRE methods that typically handle the training and memory data in a unified manner, this approach splits the training and memory data of each task into two parts respectively based on the correctness of the initial responses and treats them differently through dual-task fine-tuning. In addition, leveraging the advantages of LLM's instruction-following ability, we propose a novel instruction-based contrastive tuning strategy for LLM to continuously correct current cognitive biases with the guidance of previous data in an instruction-tuning manner, which mitigates the gap between old and new relations in a more suitable way for LLMs. We experimentally evaluate our model on TACRED and FewRel, and the results show that our model achieves new state-of-the-art CRE performance with significant improvements, demonstrating the importance of specializing in exploiting error cases.
Abstract (translated)
持续关系抽取（CRE）的目标是不断学习新出现的关系，同时避免灾难性遗忘。现有的CRE方法主要采用记忆回放和对比学习来减轻灾难性遗忘问题。然而，这些方法并未充分重视能够更有效地揭示模型认知偏差的错误案例。为解决这一问题，我们提出了一种基于指令的持续对比微调方法，适用于大型语言模型（LLMs）在CRE中的应用。与现有的CRE方法通常以统一方式处理训练和记忆数据不同，该方法根据初始响应的正确性将每个任务的训练数据和记忆数据分别划分为两部分，并通过双任务微调进行区别对待。此外，利用大型语言模型遵循指令的优势，我们提出了一种新颖的基于指令的对比微调策略，使LLM能够持续地在指令调优模式下用先前的数据指导纠正当前的认知偏差，以更符合LLMs特点的方式缓解旧关系和新关系之间的差距。我们在TACRED和FewRel数据集上实验验证了我们的模型，并且结果显示，与现有方法相比，我们的模型取得了新的最优CRE性能，证明了专注于利用错误案例的重要性。
URL
https://arxiv.org/abs/2508.12031
PDF
-
MCP-Orchestrated Multi-Agent System for Automated Disinformation Detection
2025-08-13 19:14:48Alexandru-Andrei Avram, Adrian Groza, Alexandru LecuAbstract
The large spread of disinformation across digital platforms creates significant challenges to information integrity. This paper presents a multi-agent system that uses relation extraction to detect disinformation in news articles, focusing on titles and short text snippets. The proposed Agentic AI system combines four agents: (i) a machine learning agent (logistic regression), (ii) a Wikipedia knowledge check agent (which relies on named entity recognition), (iii) a coherence detection agent (using LLM prompt engineering), and (iv) a web-scraped data analyzer that extracts relational triplets for fact checking. The system is orchestrated via the Model Context Protocol (MCP), offering shared context and live learning across components. Results demonstrate that the multi-agent ensemble achieves 95.3% accuracy with an F1 score of 0.964, significantly outperforming individual agents and traditional approaches. The weighted aggregation method, mathematically derived from individual agent misclassification rates, proves superior to algorithmic threshold optimization. The modular architecture makes the system easily scalable, while also maintaining details of the decision processes.
Abstract (translated)
数字平台上虚假信息的广泛传播对信息完整性构成了重大挑战。本文提出了一种多代理系统，该系统利用关系抽取技术来检测新闻文章中的虚假信息，重点分析标题和简短的文字片段。提出的“代理式AI”系统结合了四个代理：(i) 一个机器学习代理（逻辑回归），(ii) 一个维基百科知识检查代理（依赖于命名实体识别），(iii) 一个连贯性检测代理（使用大语言模型提示工程），以及(iv) 一个网页抓取数据分析器，用于提取关系三元组以进行事实核查。该系统通过“模型上下文协议”（MCP）来协调各个部分，提供共享的上下文和实时学习功能。 实验结果显示，多代理系统的整体准确率达到95.3%，F1值为0.964，显著优于单一代理和传统方法。通过计算每个代理错误分类率得出的加权聚合方法证明比算法阈值优化更有效。模块化架构使该系统易于扩展，并且能够详细记录决策过程。 这种多代理系统的提出不仅在技术上解决了虚假信息检测的问题，而且展示了如何利用不同类型的智能代理协同工作以提高系统的整体性能和准确性。
URL
https://arxiv.org/abs/2508.10143
PDF
-
Data-Efficient Biomedical In-Context Learning: A Diversity-Enhanced Submodular Perspective
2025-08-11 16:13:21Jun Wang, Zaifu Zhan, Qixin Zhang, Mingquan Lin, Meijia Song, Rui ZhangAbstract
Recent progress in large language models (LLMs) has leveraged their in-context learning (ICL) abilities to enable quick adaptation to unseen biomedical NLP tasks. By incorporating only a few input-output examples into prompts, LLMs can rapidly perform these new tasks. While the impact of these demonstrations on LLM performance has been extensively studied, most existing approaches prioritize representativeness over diversity when selecting examples from large corpora. To address this gap, we propose Dual-Div, a diversity-enhanced data-efficient framework for demonstration selection in biomedical ICL. Dual-Div employs a two-stage retrieval and ranking process: First, it identifies a limited set of candidate examples from a corpus by optimizing both representativeness and diversity (with optional annotation for unlabeled data). Second, it ranks these candidates against test queries to select the most relevant and non-redundant demonstrations. Evaluated on three biomedical NLP tasks (named entity recognition (NER), relation extraction (RE), and text classification (TC)) using LLaMA 3.1 and Qwen 2.5 for inference, along with three retrievers (BGE-Large, BMRetriever, MedCPT), Dual-Div consistently outperforms baselines-achieving up to 5% higher macro-F1 scores-while demonstrating robustness to prompt permutations and class imbalance. Our findings establish that diversity in initial retrieval is more critical than ranking-stage optimization, and limiting demonstrations to 3-5 examples maximizes performance efficiency.
Abstract (translated)
最近在大型语言模型（LLMs）方面的进展，利用了它们的上下文学习（ICL）能力来快速适应未见过的生物医学自然语言处理任务。通过将少量输入输出示例整合到提示中，LLMs能够迅速执行这些新任务。尽管已经对这些示范如何影响LLM性能进行了广泛研究，但现有方法在从大型语料库中选择示例时通常优先考虑代表性而非多样性。为了解决这一空白，我们提出了Dual-Div，这是一种增强多样性的高效数据框架，用于生物医学ICL中的示范选择。Dual-Div采用两阶段的检索和排序过程：首先，通过优化代表性和多样性（可选地针对未标记数据进行注释），它从语料库中识别出一组有限的候选示例；其次，在测试查询的基础上对这些候选者进行排名，以选择最相关且非冗余的示范。在三项生物医学NLP任务（命名实体识别(NER)、关系抽取(RE)和文本分类(TC)）上进行了评估，使用LLaMA 3.1 和 Qwen 2.5 进行推理，并结合了三种检索器（BGE-Large、BMRetriever、MedCPT），Dual-Div始终超越基线方法，在所有任务中宏F1分数提高了高达5%，同时展示了对提示排列和类别不平衡的鲁棒性。我们的研究结果表明，初始检索阶段中的多样性比排序阶段优化更为关键，并且将示范限制为3-5个示例可以最大化性能效率。
URL
https://arxiv.org/abs/2508.08140
PDF
