Abstract
Human annotation is central to NLP evaluation, yet subjective tasks often exhibit substantial variability across annotators. While large language models (LLMs) can provide structured reasoning to support annotation, their influence on human annotation behavior remains unclear. We introduce ReasonAlign, a reasoning-based annotation scaffold that exposes LLM-generated explanations while withholding predicted labels. We frame this as a controlled study of how reasoning affects human annotation behavior, rather than a full evaluation of annotation accuracy. Using a two-pass protocol inspired by Delphi-style revision, annotators first label instances independently and then revise their decisions after viewing model-generated reasoning. We evaluate the approach on sentiment classification and opinion detection tasks, analyzing changes in inter-annotator agreement and revision behavior. To quantify these effects, we introduce the Annotator Effort Proxy (AEP), a metric capturing the proportion of labels revised after exposure to reasoning. Our results show that exposure to reasoning is associated with increased agreement alongside minimal revision, suggesting that reasoning primarily helps resolve ambiguous cases without inducing widespread changes. These findings provide insight into how reasoning explanations shape annotation consistency and highlight reasoning-based scaffolds as a practical mechanism for supporting human-AI annotation workflows.
Abstract (translated)
人类标注是自然语言处理评估的核心环节，但主观性任务在标注者间常存在显著差异。尽管大语言模型能提供结构化推理以辅助标注，但其对人类标注行为的影响尚不明确。我们提出ReasonAlign——一种基于推理的标注支架，该支架展示大语言模型生成的解释但隐藏预测标签。我们将此设计为一项受控研究，旨在探究推理如何影响人类标注行为，而非对标注准确性进行全面评估。研究采用受德尔斐式修订启发的双轮协议：标注者首先独立标注实例，随后在查看模型生成的推理后修订决策。我们在情感分类与观点检测任务中评估该方法，分析标注者间一致性与修订行为的变化。为量化这些效应，我们引入标注者努力代理指标（AEP），该指标捕捉标注者在接触推理后修订标签的比例。结果表明，接触推理与更高的一致性及最小的修订行为相关，暗示推理主要帮助解决模糊案例，而未引发广泛改动。这些发现揭示了推理解释如何塑造标注一致性，并凸显基于推理的支架可作为支持人机协同标注流程的实用机制。
URL
https://arxiv.org/abs/2603.21094
