Abstract
We present MarkMatch, a retrieval system for detecting whether two paper ballot marks were filled by the same hand. Unlike the previous SOTA method BubbleSig, which used binary classification on isolated mark pairs, MarkMatch ranks stylistic similarity between a query mark and a mark in the database using contrastive learning. Our model is trained with a dense batch similarity matrix and a dual loss objective. Each sample is contrasted against many negatives within each batch, enabling the model to learn subtle handwriting difference and improve generalization under handwriting variation and visual noise, while diagonal supervision reinforces high confidence on true matches. The model achieves an F1 score of 0.943, surpassing BubbleSig's best performance. MarkMatch also integrates Segment Anything Model for flexible mark extraction via box- or point-based prompts. The system offers election auditors a practical tool for visual, non-biometric investigation of suspicious ballots.
Abstract (translated)
我们介绍了MarkMatch，这是一种检索系统，用于检测两张选票上的标记是否由同一人填写。与之前最先进的方法BubbleSig不同，后者使用二分类方式处理孤立的标记对，MarkMatch则通过对比学习来评估查询标记和数据库中一个标记之间的风格相似度。我们的模型在密集批次相似性矩阵上进行训练，并采用双重损失目标函数。每个样本都与其他许多负例（不匹配的例子）进行对比，在同一批量内进行训练，使模型能够捕捉到细微的手写差异并提高其在手写变化和视觉噪声下的泛化能力。对角监督则确保了真实匹配时的高度信心。该模型达到了0.943的F1分数，超过了BubbleSig的最佳性能记录。 MarkMatch还集成了Segment Anything Model（SAM），通过基于框或点的提示灵活提取标记。这一系统为选举审计员提供了一个实用工具，可以进行非生物识别、视觉检查可疑选票的方法。
URL
https://arxiv.org/abs/2505.07032
