Abstract
Structured information extraction from scientific literature is crucial for capturing core concepts and emerging trends in specialized fields. While existing datasets aid model development, most focus on specific publication sections due to domain complexity and the high cost of annotating scientific texts. To address this limitation, we introduce SciNLP - a specialized benchmark for full-text entity and relation extraction in the Natural Language Processing (NLP) domain. The dataset comprises 60 manually annotated full-text NLP publications, covering 7,072 entities and 1,826 relations. Compared to existing research, SciNLP is the first dataset providing full-text annotations of entities and their relationships in the NLP domain. To validate the effectiveness of SciNLP, we conducted comparative experiments with similar datasets and evaluated the performance of state-of-the-art supervised models on this dataset. Results reveal varying extraction capabilities of existing models across academic texts of different lengths. Cross-comparisons with existing datasets show that SciNLP achieves significant performance improvements on certain baseline models. Using models trained on SciNLP, we implemented automatic construction of a fine-grained knowledge graph for the NLP domain. Our KG has an average node degree of 3.2 per entity, indicating rich semantic topological information that enhances downstream applications. The dataset is publicly available at this https URL.
Abstract (translated)
从科学文献中提取结构化信息对于捕捉特定领域的核心概念和新兴趋势至关重要。尽管现有的数据集有助于模型开发，但大多数现有数据集由于领域复杂性和标注科学文本的高昂成本而仅侧重于具体出版物的部分内容。为了克服这一限制，我们引入了SciNLP——这是一个专门用于自然语言处理（NLP）领域全文实体与关系提取的专业基准测试。该数据集包括60篇手写的、完整注释的NLP出版物，覆盖7,072个实体和1,826种关系。相比于现有研究，SciNLP是首个提供NLP领域内全文本中的实体及其关系标注的数据集。 为了验证SciNLP的有效性，我们进行了与类似数据集的比较实验，并评估了最先进的监督模型在该数据集上的性能表现。结果表明，现有的模型对于不同长度学术文本的信息抽取能力存在差异。交叉对比现有数据集显示，SciNLP对某些基准模型实现了显著的性能提升。 利用基于SciNLP训练的模型，我们自动构建了一个针对NLP领域的细粒度知识图谱（KG）。我们的KG每个实体平均有3.2个关联节点，显示出丰富的语义拓扑信息，这有助于下游应用的发展。该数据集在以下链接公开可得：[提供URL]
URL
https://arxiv.org/abs/2509.07801
