Abstract
The rapid expansion of biomedical literature has heightened the need for scalable natural language processing (NLP) solutions. While GPT-4 substantially narrowed the gap with task-specific systems, especially in question answering, its performance across other domains remained uneven. We updated a standardized BioNLP benchmark to evaluate GPT-5 and GPT-4o under zero-, one-, and five-shot prompting across 12 datasets spanning six task families: named entity recognition, relation extraction, multi-label document classification, question answering, text summarization, and text simplification. Using fixed prompt templates, identical decoding parameters, and batch inference, we report primary metrics per dataset and include prior results for GPT-4, GPT-3.5, and LLaMA-2-13B for comparison. GPT-5 achieved the strongest overall benchmark performance, with macro-average scores rising to 0.557 under five-shot prompting versus 0.506 for GPT-4 and 0.508 for GPT-4o. On MedQA, GPT-5 reached 94.1% accuracy, exceeding the previous supervised state of the art by over fifty points, and attained parity with supervised systems on PubMedQA (0.734). In extraction tasks, GPT-5 delivered major gains in chemical NER (0.886 F1) and ChemProt relation extraction (0.616 F1), outperforming GPT-4 and GPT-4o, though summarization and disease NER still lagged behind domain-specific baselines. These results establish GPT-5 as a general-purpose model now offering deployment-ready performance for reasoning-oriented biomedical QA, while precision-critical extraction and evidence-dense summarization continue to favor fine-tuned or hybrid approaches. The benchmark delineates where simple prompting suffices and where retrieval-augmented or planning-based scaffolds are likely required, providing actionable guidance for BioNLP system design as frontier models advance.
Abstract (translated)
生物医学文献的快速扩张增强了对可扩展自然语言处理（NLP）解决方案的需求。虽然GPT-4在特定任务系统方面，特别是在问题回答上大大缩小了差距，但在其他领域的表现仍然参差不齐。我们更新了一个标准化的BioNLP基准测试，以评估GPT-5和GPT-4o在零样本、一拍和五拍提示下跨12个数据集的表现，这些数据集涵盖了六个任务家族：命名实体识别、关系抽取、多标签文档分类、问题回答、文本摘要和文本简化。使用固定的提示模板、相同的解码参数以及批量推理，我们报告了每个数据集的主要指标，并包含了GPT-4、GPT-3.5和LLaMA-2-13B的先前结果进行比较。 在五拍提示下，GPT-5取得了最强的整体基准性能，宏平均分达到了0.557，相比之下，GPT-4为0.506，而GPT-4o为0.508。在MedQA上，GPT-5的准确率达到了94.1%，超过了之前的监督学习最优状态超过五十个百分点，并且在PubMedQA（0.734）中与监督系统持平。在抽取任务中，GPT-5在化学实体识别（NER）方面取得了显著提升（F1为0.886），以及ChemProt关系提取（F1为0.616），超过了GPT-4和GPT-4o的表现；尽管摘要生成和疾病实体识别仍落后于领域特定基准。这些结果表明，GPT-5已成为一种通用模型，现在可以提供适用于以推理为导向的生物医学问答部署就绪性能，而精度关键的提取以及证据密集型总结任务仍然更倾向于微调或混合方法。 该基准测试明确了简单的提示在哪些情况下足够使用，在哪些情况下可能需要检索增强或基于计划的框架，并为BioNLP系统设计提供了可操作性的指导，随着前沿模型的进步。
URL
https://arxiv.org/abs/2509.04462
