Abstract
Assessing student handwritten scratchwork is crucial for personalized educational feedback but presents unique challenges due to diverse handwriting, complex layouts, and varied problem-solving approaches. Existing educational NLP primarily focuses on textual responses and neglects the complexity and multimodality inherent in authentic handwritten scratchwork. Current multimodal large language models (MLLMs) excel at visual reasoning but typically adopt an "examinee perspective", prioritizing generating correct answers rather than diagnosing student errors. To bridge these gaps, we introduce ScratchMath, a novel benchmark specifically designed for explaining and classifying errors in authentic handwritten mathematics scratchwork. Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students, supporting two key tasks: Error Cause Explanation (ECE) and Error Cause Classification (ECC), with seven defined error types. The dataset is meticulously annotated through rigorous human-machine collaborative approaches involving multiple stages of expert labeling, review, and verification. We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts, especially in visual recognition and logical reasoning. Proprietary models notably outperform open-source models, with large reasoning models showing strong potential for error explanation. All evaluation data and frameworks are publicly available to facilitate further research.
Abstract (translated)
评估学生手写草稿演算对个性化教育反馈至关重要，但由于字迹多样、布局复杂及解题思路各异，该任务面临独特挑战。现有教育自然语言处理研究主要聚焦文本作答，忽视了真实手写草稿演算中固有的复杂性与多模态特性。当前多模态大语言模型虽擅长视觉推理，但通常采用"应试视角"，优先生成正确答案而非诊断学生错误。为弥补这些差距，我们推出ScratchMath——首个专为解释和分类真实手写数学草稿演算错误而设计的基准数据集。该数据集包含1720份中国中小学生的数学样本，支持错误归因解释（ECE）与错误归因分类（ECC）两项核心任务，涵盖七类明确定义的错误类型。数据集通过专家标注、复核与验证的多阶段人机协作流程精心标注。我们在ScratchMath上系统评估了16个主流多模态大语言模型，发现其与人类专家存在显著性能差距，尤其在视觉识别与逻辑推理方面。专有模型普遍优于开源模型，大型推理模型在错误解释方面展现强劲潜力。所有评估数据与框架均已公开，以推动后续研究。
URL
https://arxiv.org/abs/2603.24961
