Handwriting
-
An Improved U-Net Model for Offline handwriting signature denoising
Abstract
Handwriting signatures, as an important means of identity recognition, are widely used in multiple fields such as financial transactions, commercial contracts and personal affairs due to their legal effect and uniqueness. In forensic science appraisals, the analysis of offline handwriting signatures requires the appraiser to provide a certain number of signature samples, which are usually derived from various historical contracts or archival materials. However, the provided handwriting samples are often mixed with a large amount of interfering information, which brings severe challenges to handwriting identification work. This study proposes a signature handwriting denoising model based on the improved U-net structure, aiming to enhance the robustness of the signature recognition system. By introducing discrete wavelet transform and PCA transform, the model's ability to suppress noise has been enhanced. The experimental results show that this modelis significantly superior to the traditional methods in denoising effect, can effectively improve the clarity and readability of the signed images, and provide more reliable technical support for signature analysis and recognition.
Abstract (translated)
手写签名作为一种重要的身份识别手段，在金融交易、商业合同和个人事务等多个领域中广泛应用，因其法律效力和独特性而备受重视。在法医科学鉴定中，离线手写签名的分析要求评估人员提供一定数量的手写签名样本，这些样本通常从各种历史合同或档案材料中提取。然而，提供的书写样本往往掺杂了大量的干扰信息，这对手写识别工作带来了严峻挑战。本研究提出了一种基于改进U-net结构的手写签名去噪模型，旨在增强签名识别系统的鲁棒性。通过引入离散小波变换和主成分分析（PCA）变换，该模型的降噪能力得到了提升。实验结果显示，相比传统方法，此模型在去噪效果上具有显著优势，能够有效提高签名字迹的清晰度和可读性，并为签名分析与识别提供了更加可靠的技术支持。
URL
https://arxiv.org/abs/2507.00365
PDF
-
A Transformer Based Handwriting Recognition System Jointly Using Online and Offline Features
2025-06-25 08:58:47Ayush Lodh, Ritabrata Chakraborty, Shivakumara Palaiahnakote, Umapada PalAbstract
We posit that handwriting recognition benefits from complementary cues carried by the rasterized complex glyph and the pen's trajectory, yet most systems exploit only one modality. We introduce an end-to-end network that performs early fusion of offline images and online stroke data within a shared latent space. A patch encoder converts the grayscale crop into fixed-length visual tokens, while a lightweight transformer embeds the $(x, y, \text{pen})$ sequence. Learnable latent queries attend jointly to both token streams, yielding context-enhanced stroke embeddings that are pooled and decoded under a cross-entropy loss objective. Because integration occurs before any high-level classification, temporal cues reinforce each other during representation learning, producing stronger writer independence. Comprehensive experiments on IAMOn-DB and VNOn-DB demonstrate that our approach achieves state-of-the-art accuracy, exceeding previous bests by up to 1\%. Our study also shows adaptation of this pipeline with gesturification on the ISI-Air dataset. Our code can be found here.
Abstract (translated)
我们提出，手写识别可以从栅格化复杂字符和笔迹轨迹所携带的互补线索中获益，然而大多数系统仅利用了一种模态。为此，我们引入了一个端到端网络，在共享潜在空间内对离线图像与在线笔画数据进行早期融合。一个补丁编码器将灰度裁剪部分转换为固定长度的视觉令牌，而轻量级变压器则嵌入了$(x, y, \text{pen})$序列。可学习的潜在查询同时关注两个令牌流，生成增强上下文的笔画嵌入，并在交叉熵损失目标下进行池化和解码。由于这种集成发生在任何高层次分类之前，在表示学习过程中时间线索相互强化，从而产生更强的书写者独立性。 我们在IAMOn-DB和VNOn-DB数据集上进行了全面实验，结果表明我们的方法实现了业界领先精度，比先前的最佳模型高出最多1%。此外，研究表明在ISI-Air数据集中使用手势化技术也可以适应该管道的应用。我们的代码可以在这里找到。
URL
https://arxiv.org/abs/2506.20255
PDF
-
Unfolding the Past: A Comprehensive Deep Learning Approach to Analyzing Incunabula Pages
2025-06-22 15:33:20Klaudia Ropel, Krzysztof Kutt, Luiz do Valle Miranda, Grzegorz J. NalepaAbstract
We developed a proof-of-concept method for the automatic analysis of the structure and content of incunabula pages. A custom dataset comprising 500 annotated pages from five different incunabula was created using resources from the Jagiellonian Digital Library. Each page was manually labeled with five predefined classes: Text, Title, Picture, Table, and Handwriting. Additionally, the publicly available DocLayNet dataset was utilized as supplementary training data. To perform object detection, YOLO11n and YOLO11s models were employed and trained using two strategies: a combined dataset (DocLayNet and the custom dataset) and the custom dataset alone. The highest performance (F1 = 0.94) was achieved by the YOLO11n model trained exclusively on the custom data. Optical character recognition was then conducted on regions classified as Text, using both Tesseract and Kraken OCR, with Tesseract demonstrating superior results. Subsequently, image classification was applied to the Picture class using a ResNet18 model, achieving an accuracy of 98.7% across five subclasses: Decorative_letter, Illustration, Other, Stamp, and Wrong_detection. Furthermore, the CLIP model was utilized to generate semantic descriptions of illustrations. The results confirm the potential of machine learning in the analysis of early printed books, while emphasizing the need for further advancements in OCR performance and visual content interpretation.
Abstract (translated)
我们开发了一种概念验证方法，用于自动分析古籍印刷品页面的结构和内容。使用来自雅盖隆数字图书馆（Jagiellonian Digital Library）的资源创建了一个包含500页注释数据集，这些注释来自于五本不同的古登堡圣经，每一页都被手动标注为五个预定义类别之一：文本、标题、图片、表格和手写。此外，还使用了公开可用的DocLayNet数据集作为辅助训练数据。 为了执行目标检测任务，我们采用了YOLO11n和YOLO11s模型，并通过两种策略对其进行训练：一种是结合使用的数据集（DocLayNet和自定义数据集），另一种则是仅使用自定义数据集。结果显示，专门在自定义数据集上进行训练的YOLO11n模型取得了最佳性能（F1分数为0.94）。 随后，我们对被分类为“文本”的区域进行了光学字符识别（OCR）。这项任务采用了Tesseract和Kraken两种OCR工具，其中Tesseract表现更优。接着，我们使用ResNet18模型对图片类别进行图像分类，在五种子类（装饰字母、插图、其他、印章以及错误检测）中实现了98.7%的准确率。最后，还利用了CLIP模型生成关于插图的语义描述。 这些结果证实了机器学习在分析早期印刷书籍方面的潜力，同时也强调了需要进一步提高OCR性能和视觉内容解读方面的发展需求。
URL
https://arxiv.org/abs/2506.18069
PDF
-
Learning to Align: Addressing Character Frequency Distribution Shifts in Handwritten Text Recognition
2025-06-11 15:20:30Panagiotis Kaliosis, John PavlopoulosAbstract
Handwritten text recognition aims to convert visual input into machine-readable text, and it remains challenging due to the evolving and context-dependent nature of handwriting. Character sets change over time, and character frequency distributions shift across historical periods or regions, often causing models trained on broad, heterogeneous corpora to underperform on specific subsets. To tackle this, we propose a novel loss function that incorporates the Wasserstein distance between the character frequency distribution of the predicted text and a target distribution empirically derived from training data. By penalizing divergence from expected distributions, our approach enhances both accuracy and robustness under temporal and contextual intra-dataset shifts. Furthermore, we demonstrate that character distribution alignment can also improve existing models at inference time without requiring retraining by integrating it as a scoring function in a guided decoding scheme. Experimental results across multiple datasets and architectures confirm the effectiveness of our method in boosting generalization and performance. We open source our code at this https URL.
Abstract (translated)
手写文本识别旨在将视觉输入转换为机器可读的文本，但由于手写的演变和上下文依赖性，这一过程仍然具有挑战性。字符集会随时间变化，不同历史时期或地区的字符频率分布也会有所不同，这通常会导致在广泛、异质语料库上训练的模型在其特定子集中表现不佳。为此，我们提出了一种新的损失函数，该函数结合了预测文本中的字符频率分布与从训练数据中经验性推导出的目标分布之间的Wasserstein距离。通过惩罚偏离预期分布的行为，我们的方法可以提高模型在时间变化和上下文内的数据集内部变化情况下的准确性和鲁棒性。此外，我们还证明，在推理阶段将字符分布对齐整合为指导解码方案中的评分函数，可以在不重新训练的情况下改善现有模型的性能。通过多个数据集和架构上的实验结果证实了我们的方法在提高泛化能力和性能方面的有效性。我们将源代码开源于此 [URL]（请用实际链接替换）。
URL
https://arxiv.org/abs/2506.09846
PDF
-
Creating a Historical Migration Dataset from Finnish Church Records, 1800-1920
2025-06-09 17:32:55Ari Vesalainen, Jenna Kanerva, Aida Nitsch, Kiia Korsu, Ilari Larkiola, Laura Ruotsalainen, Filip GinterAbstract
This article presents a large-scale effort to create a structured dataset of internal migration in Finland between 1800 and 1920 using digitized church moving records. These records, maintained by Evangelical-Lutheran parishes, document the migration of individuals and families and offer a valuable source for studying historical demographic patterns. The dataset includes over six million entries extracted from approximately 200,000 images of handwritten migration records. The data extraction process was automated using a deep learning pipeline that included layout analysis, table detection, cell classification, and handwriting recognition. The complete pipeline was applied to all images, resulting in a structured dataset suitable for research. The dataset can be used to study internal migration, urbanization, and family migration, and the spread of disease in preindustrial Finland. A case study from the Elimäki parish shows how local migration histories can be reconstructed. The work demonstrates how large volumes of handwritten archival material can be transformed into structured data to support historical and demographic research.
Abstract (translated)
本文介绍了一项大规模的努力，利用数字化的教会迁移记录创建了1800年至1920年间芬兰国内移民的结构化数据集。这些记录由福音路德会教区维护，详细记载了个人和家庭的迁移情况，是研究历史人口模式的重要资料来源。该数据集包含超过六百万条从大约二十万张手写迁移记录图片中提取出来的信息。 数据提取过程采用了深度学习流水线自动化处理，包括版面分析、表格检测、单元格分类以及手写识别等步骤。整个流水线应用于所有图像后，生成了一个适合研究的结构化数据集。该数据集可以用于研究内部移民、城市化、家庭迁移以及工业化前芬兰疾病的传播。 通过艾利马基教区的一个案例研究展示了如何重建本地迁移历史。这项工作证明了大量手写档案资料可以通过转化为结构化数据来支持历史和人口学的研究。
URL
https://arxiv.org/abs/2506.07960
PDF
-
Towards the Influence of Text Quantity on Writer Retrieval
2025-06-09 09:05:15Marco Peer, Robert Sablatnig, Florian KleberAbstract
This paper investigates the task of writer retrieval, which identifies documents authored by the same individual within a dataset based on handwriting similarities. While existing datasets and methodologies primarily focus on page level retrieval, we explore the impact of text quantity on writer retrieval performance by evaluating line- and word level retrieval. We examine three state-of-the-art writer retrieval systems, including both handcrafted and deep learning-based approaches, and analyze their performance using varying amounts of text. Our experiments on the CVL and IAM dataset demonstrate that while performance decreases by 20-30% when only one line of text is used as query and gallery, retrieval accuracy remains above 90% of full-page performance when at least four lines are included. We further show that text-dependent retrieval can maintain strong performance in low-text scenarios. Our findings also highlight the limitations of handcrafted features in low-text scenarios, with deep learning-based methods like NetVLAD outperforming traditional VLAD encoding.
Abstract (translated)
本文探讨了书写者检索任务，该任务基于手写相似性在数据集中识别由同一人创作的文档。虽然现有的数据集和方法主要集中在页面级别的检索上，我们通过评估行级和词级检索来研究文本量对书写者检索性能的影响。我们分析了三种最先进的书写者检索系统（包括手工设计和深度学习基于的方法），并使用不同数量的文本进行性能测试。我们在CVL和IAM数据集上的实验表明，当仅使用一行文本作为查询和检索库时，性能会下降20-30%，但如果至少包含四行文本，检索准确率仍然可以保持在全页表现的90%以上。此外，我们还展示了基于文本依赖性的检索可以在低文本场景中维持较强的性能。我们的研究结果进一步强调了手工设计特征在低文本环境中的局限性，表明如NetVLAD这样的深度学习方法优于传统的VLAD编码方法。
URL
https://arxiv.org/abs/2506.07566
PDF
-
Evaluating Vision-Language and Large Language Models for Automated Student Assessment in Indonesian Classrooms
2025-06-05 09:41:09Nurul Aisyah, Muhammad Dehan Al Kautsar, Arif Hidayat, Raqib Chowdhury, Fajri KotoAbstract
Although vision-language and large language models (VLM and LLM) offer promising opportunities for AI-driven educational assessment, their effectiveness in real-world classroom settings, particularly in underrepresented educational contexts, remains underexplored. In this study, we evaluated the performance of a state-of-the-art VLM and several LLMs on 646 handwritten exam responses from grade 4 students in six Indonesian schools, covering two subjects: Mathematics and English. These sheets contain more than 14K student answers that span multiple choice, short answer, and essay questions. Assessment tasks include grading these responses and generating personalized feedback. Our findings show that the VLM often struggles to accurately recognize student handwriting, leading to error propagation in downstream LLM grading. Nevertheless, LLM-generated feedback retains some utility, even when derived from imperfect input, although limitations in personalization and contextual relevance persist.
Abstract (translated)
尽管视觉-语言模型（VLM）和大型语言模型（LLM）为基于AI的教育评估提供了许多机会，但它们在现实世界课堂环境中的有效性，尤其是在教育资源不足的情境下，仍然有待深入研究。在这项研究中，我们评估了一种最先进的VLM以及几种LLM在印度尼西亚六所学校的646份四年级学生手写考试答案上的表现，涵盖数学和英语两门学科。这些答卷包含超过1.4万个学生的答题内容，包括多项选择题、简答题和作文题。评估任务包括对这些回答进行评分并生成个性化反馈。我们的研究发现表明，VLM在准确识别学生手写方面经常遇到困难，这会导致下游LLM评分级别的错误传播。尽管如此，在基于不完美的输入数据时，由LLM生成的反馈仍具有一定的实用性，但个性化和情境相关性的局限性仍然存在。
URL
https://arxiv.org/abs/2506.04822
PDF
-
Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition
2025-05-29 15:41:00Yu Li, Jin Jiang, Jianhua Zhu, Shuai Peng, Baole Wei, Yuxuan Zhou, Liangcai GaoAbstract
Handwritten Mathematical Expression Recognition (HMER) remains a persistent challenge in Optical Character Recognition (OCR) due to the inherent freedom of symbol layout and variability in handwriting styles. Prior methods have faced performance bottlenecks, proposing isolated architectural modifications that are difficult to integrate coherently into a unified framework. Meanwhile, recent advances in pretrained vision-language models (VLMs) have demonstrated strong cross-task generalization, offering a promising foundation for developing unified solutions. In this paper, we introduce Uni-MuMER, which fully fine-tunes a VLM for the HMER task without modifying its architecture, effectively injecting domain-specific knowledge into a generalist framework. Our method integrates three data-driven tasks: Tree-Aware Chain-of-Thought (Tree-CoT) for structured spatial reasoning, Error-Driven Learning (EDL) for reducing confusion among visually similar characters, and Symbol Counting (SC) for improving recognition consistency in long expressions. Experiments on the CROHME and HME100K datasets show that Uni-MuMER achieves new state-of-the-art performance, surpassing the best lightweight specialized model SSAN by 16.31% and the top-performing VLM Gemini2.5-flash by 24.42% in the zero-shot setting. Our datasets, models, and code are open-sourced at: this https URL
Abstract (translated)
手写数学表达式识别（HMER）在光学字符识别（OCR）中仍然是一项持续的挑战，这是由于符号布局的自由度和书写风格的变化所致。先前的方法由于提出孤立的架构修改而遇到了性能瓶颈，这些修改难以整合到一个统一框架内。与此同时，近期预训练视觉语言模型（VLMs）在跨任务泛化方面表现出色，为开发统一体解决方案提供了有前景的基础。本文中，我们介绍了一种名为Uni-MuMER的方法，该方法无需修改其架构即可对VLM进行全面微调，从而有效地将领域特定知识注入到通用框架中。我们的方法整合了三项数据驱动的任务：结构化空间推理的树感知链式思维（Tree-CoT）、减少视觉相似字符混淆的错误驱动学习（EDL）以及提高长表达式识别一致性的符号计数（SC）。在CROHME和HME100K数据集上的实验表明，Uni-MuMER达到了新的最先进的性能，在零样本设置下，其表现优于最轻量级的专业模型SSAN 16.31%，优于顶级的VLM Gemini2.5-flash 24.42%。 我们的数据集、模型和代码已开源：[请在此处插入链接]（请注意原文中的具体网址无法直接显示，请访问原论文获取准确的URL）。
URL
https://arxiv.org/abs/2505.23566
PDF
-
MetaWriter: Personalized Handwritten Text Recognition Using Meta-Learned Prompt Tuning
2025-05-26 20:26:16Wenhao Gu, Li Gu, Ching Yee Suen, Yang WangAbstract
Recent advancements in handwritten text recognition (HTR) have enabled the effective conversion of handwritten text to digital formats. However, achieving robust recognition across diverse writing styles remains challenging. Traditional HTR methods lack writer-specific personalization at test time due to limitations in model architecture and training strategies. Existing attempts to bridge this gap, through gradient-based meta-learning, still require labeled examples and suffer from parameter-inefficient fine-tuning, leading to substantial computational and memory overhead. To overcome these challenges, we propose an efficient framework that formulates personalization as prompt tuning, incorporating an auxiliary image reconstruction task with a self-supervised loss to guide prompt adaptation with unlabeled test-time examples. To ensure self-supervised loss effectively minimizes text recognition error, we leverage meta-learning to learn the optimal initialization of the prompts. As a result, our method allows the model to efficiently capture unique writing styles by updating less than 1% of its parameters and eliminating the need for time-intensive annotation processes. We validate our approach on the RIMES and IAM Handwriting Database benchmarks, where it consistently outperforms previous state-of-the-art methods while using 20x fewer parameters. We believe this represents a significant advancement in personalized handwritten text recognition, paving the way for more reliable and practical deployment in resource-constrained scenarios.
Abstract (translated)
最近在手写文本识别（HTR）领域的进展使得将手写文本转换为数字格式成为可能。然而，实现不同书写风格下的稳健识别仍然具有挑战性。传统的方法由于模型架构和训练策略的限制，在测试时缺乏特定于书写的个性化调整。尽管有一些尝试通过基于梯度的元学习来弥合这一差距，但这些方法仍需要标注示例，并且因参数效率低下的微调而遭受计算量大和内存开销大的问题。 为了解决这些问题，我们提出了一种高效的框架，该框架将个性化视为提示调整，并结合了辅助图像重建任务以及自监督损失来引导未标记的测试时间实例中的提示适应。为了确保自监督损失能有效减少文本识别误差，我们利用元学习来学习提示的最佳初始设置。因此，我们的方法使模型能够通过更新不到1%的参数高效地捕捉独特的书写风格，并消除了耗时的标注过程需求。 我们在RIMES和IAM手写数据库基准测试上验证了这一方法的有效性，在这些测试中，它持续优于先前最先进的方法，同时使用的参数减少了20倍。我们认为这代表了个性化手写文本识别的重要进展，为资源受限场景中的更可靠和实际部署铺平了道路。
URL
https://arxiv.org/abs/2505.20513
PDF
-
WriteViT: Handwritten Text Generation with Vision Transformer
2025-05-19 15:17:53Dang Hoai Nam, Huynh Tong Dang Khoa, Vo Nguyen Le DuyAbstract
Humans can quickly generalize handwriting styles from a single example by intuitively separating content from style. Machines, however, struggle with this task, especially in low-data settings, often missing subtle spatial and stylistic cues. Motivated by this gap, we introduce WriteViT, a one-shot handwritten text synthesis framework that incorporates Vision Transformers (ViT), a family of models that have shown strong performance across various computer vision tasks. WriteViT integrates a ViT-based Writer Identifier for extracting style embeddings, a multi-scale generator built with Transformer encoder-decoder blocks enhanced by conditional positional encoding (CPE), and a lightweight ViT-based recognizer. While previous methods typically rely on CNNs or CRNNs, our design leverages transformers in key components to better capture both fine-grained stroke details and higher-level style information. Although handwritten text synthesis has been widely explored, its application to Vietnamese -- a language rich in diacritics and complex typography -- remains limited. Experiments on Vietnamese and English datasets demonstrate that WriteViT produces high-quality, style-consistent handwriting while maintaining strong recognition performance in low-resource scenarios. These results highlight the promise of transformer-based designs for multilingual handwriting generation and efficient style adaptation.
Abstract (translated)
人类可以从单一的样本中快速地概括出手写风格，通过直观地区分内容和风格。然而，机器在处理这种任务时，尤其是在数据量较少的情况下，经常难以捕捉到细微的空间和风格线索。为了弥补这一差距，我们引入了一种名为WriteViT的一次性手写文本合成框架，该框架结合了视觉变换器（Vision Transformers, ViT），这是一种在各种计算机视觉任务中表现出色的模型系列。WriteViT集成了基于ViT的Writer Identifier用于提取风格嵌入、采用Transformer编码-解码块并通过条件位置编码(CPE)增强的多尺度生成器，以及一个轻量级的基于ViT的识别器。相比之下，之前的方法通常依赖于CNN或CRNN模型，而我们的设计在关键组件中采用了变换器来更好地捕捉细节笔画特征和更高级别的风格信息。 尽管手写文本合成已经被广泛研究，但在越南语这种具有丰富音调符号和复杂版面的语言上的应用仍然有限。实验表明，在越南语和英语数据集上，WriteViT能够生成高质量且风格一致的手写字体，并在低资源场景中保持强大的识别性能。这些结果突显了基于变换器的设计在多语言手写生成和高效风格适应方面的潜力。
URL
https://arxiv.org/abs/2505.13235
PDF
-
A Study on the Refining Handwritten Font by Mixing Font Styles
2025-05-19 08:20:43Avinash Kumar, Kyeolhee Kang, Ammar ul Hassan, Jaeyoung ChoiAbstract
Handwritten fonts have a distinct expressive character, but they are often difficult to read due to unclear or inconsistent handwriting. FontFusionGAN (FFGAN) is a novel method for improving handwritten fonts by combining them with printed fonts. Our method implements generative adversarial network (GAN) to generate font that mix the desirable features of handwritten and printed fonts. By training the GAN on a dataset of handwritten and printed fonts, it can generate legible and visually appealing font images. We apply our method to a dataset of handwritten fonts and demonstrate that it significantly enhances the readability of the original fonts while preserving their unique aesthetic. Our method has the potential to improve the readability of handwritten fonts, which would be helpful for a variety of applications including document creation, letter writing, and assisting individuals with reading and writing difficulties. In addition to addressing the difficulties of font creation for languages with complex character sets, our method is applicable to other text-image-related tasks, such as font attribute control and multilingual font style transfer.
Abstract (translated)
手写字体具有独特的表现力，但由于笔迹不清楚或不一致，它们通常难以阅读。FontFusionGAN（FFGAN）是一种结合手写和印刷字体以改进手写字体的新方法。我们的方法利用生成对抗网络（GAN）来创建融合了手写和印刷字体优点的新型字体。通过在包含手写和印刷字体的数据集上训练GAN，它可以生成清晰且视觉效果吸引人的字体图像。我们将该方法应用于一个手写字体数据集，并证明它显著提升了原始字体的可读性，同时保留其独特的美学风格。 我们的方法有望提高手写字体的易读性，在文档创建、书信书写以及帮助阅读和写作有困难的人士等方面大有用处。此外，除了解决复杂字符集语言中字体创作的难题外，我们的方法还适用于其他与文本图像相关任务，例如控制字体属性及跨多种语言风格转换。
URL
https://arxiv.org/abs/2505.12834
PDF
-
MarkMatch: Same-Hand Stuffing Detection
2025-05-11 16:05:07Fei Zhao, Runlin Zhang, Chengcui Zhang, Nitesh SaxenaAbstract
We present MarkMatch, a retrieval system for detecting whether two paper ballot marks were filled by the same hand. Unlike the previous SOTA method BubbleSig, which used binary classification on isolated mark pairs, MarkMatch ranks stylistic similarity between a query mark and a mark in the database using contrastive learning. Our model is trained with a dense batch similarity matrix and a dual loss objective. Each sample is contrasted against many negatives within each batch, enabling the model to learn subtle handwriting difference and improve generalization under handwriting variation and visual noise, while diagonal supervision reinforces high confidence on true matches. The model achieves an F1 score of 0.943, surpassing BubbleSig's best performance. MarkMatch also integrates Segment Anything Model for flexible mark extraction via box- or point-based prompts. The system offers election auditors a practical tool for visual, non-biometric investigation of suspicious ballots.
Abstract (translated)
我们介绍了MarkMatch，这是一种检索系统，用于检测两张选票上的标记是否由同一人填写。与之前最先进的方法BubbleSig不同，后者使用二分类方式处理孤立的标记对，MarkMatch则通过对比学习来评估查询标记和数据库中一个标记之间的风格相似度。我们的模型在密集批次相似性矩阵上进行训练，并采用双重损失目标函数。每个样本都与其他许多负例（不匹配的例子）进行对比，在同一批量内进行训练，使模型能够捕捉到细微的手写差异并提高其在手写变化和视觉噪声下的泛化能力。对角监督则确保了真实匹配时的高度信心。该模型达到了0.943的F1分数，超过了BubbleSig的最佳性能记录。 MarkMatch还集成了Segment Anything Model（SAM），通过基于框或点的提示灵活提取标记。这一系统为选举审计员提供了一个实用工具，可以进行非生物识别、视觉检查可疑选票的方法。
URL
https://arxiv.org/abs/2505.07032
PDF
-
Design description of Wisdom Computing Persperctive
2025-05-02 07:12:10TianYi YuAbstract
This course design aims to develop and research a handwriting matrix recognition and step-by-step visual calculation process display system, addressing the issue of abstract formulas and complex calculation steps that students find difficult to understand when learning mathematics. By integrating artificial intelligence with visualization animation technology, the system enhances precise recognition of handwritten matrix content through the introduction of Mamba backbone networks, completes digital extraction and matrix reconstruction using the YOLO model, and simultaneously combines CoordAttention coordinate attention mechanisms to improve the accurate grasp of character spatial positions. The calculation process is demonstrated frame by frame through the Manim animation engine, vividly showcasing each mathematical calculation step, helping students intuitively understand the intrinsic logic of mathematical operations. Through dynamically generating animation processes for different computational tasks, the system exhibits high modularity and flexibility, capable of generating various mathematical operation examples in real-time according to student needs. By innovating human-computer interaction methods, it brings mathematical calculation processes to life, helping students bridge the gap between knowledge and understanding on a deeper level, ultimately achieving a learning experience where "every step is understood." The system's scalability and interactivity make it an intuitive, user-friendly, and efficient auxiliary tool in education.
Abstract (translated)
本课程设计旨在开发并研究一种手写矩阵识别与分步可视化计算过程展示系统，解决学生在学习数学时遇到的抽象公式和复杂计算步骤难以理解的问题。通过将人工智能与可视化动画技术相结合，该系统利用Mamba骨干网络提高了对手写矩阵内容的精确识别能力；采用YOLO模型完成数字提取及矩阵重构，并结合CoordAttention坐标注意力机制来提高对字符空间位置的准确把握。使用Manim动画引擎逐帧展示计算过程，生动地呈现每个数学运算步骤，帮助学生直观理解数学操作的内在逻辑。 该系统通过为不同的计算任务动态生成动画流程，在保持高度模块化和灵活性的同时能够根据学生的需要实时生成各种数学操作示例。通过创新人机交互方法，使数学计算过程更加生动有趣，有助于学生在更深层面建立知识与理解之间的桥梁，最终实现“每一步都能理解”的学习体验。系统的可扩展性和互动性使其成为一种直观、用户友好且高效的教育辅助工具。
URL
https://arxiv.org/abs/2505.03800
PDF
-
A Methodological and Structural Review of Parkinsons Disease Detection Across Diverse Data Modalities
2025-05-01 13:47:45Abu Saleh Musa Miah, taro Suzuki, Jungpil ShinAbstract
Parkinsons Disease (PD) is a progressive neurological disorder that primarily affects motor functions and can lead to mild cognitive impairment (MCI) and dementia in its advanced stages. With approximately 10 million people diagnosed globally 1 to 1.8 per 1,000 individuals, according to reports by the Japan Times and the Parkinson Foundation early and accurate diagnosis of PD is crucial for improving patient outcomes. While numerous studies have utilized machine learning (ML) and deep learning (DL) techniques for PD recognition, existing surveys are limited in scope, often focusing on single data modalities and failing to capture the potential of multimodal approaches. To address these gaps, this study presents a comprehensive review of PD recognition systems across diverse data modalities, including Magnetic Resonance Imaging (MRI), gait-based pose analysis, gait sensory data, handwriting analysis, speech test data, Electroencephalography (EEG), and multimodal fusion techniques. Based on over 347 articles from leading scientific databases, this review examines key aspects such as data collection methods, settings, feature representations, and system performance, with a focus on recognition accuracy and robustness. This survey aims to serve as a comprehensive resource for researchers, providing actionable guidance for the development of next generation PD recognition systems. By leveraging diverse data modalities and cutting-edge machine learning paradigms, this work contributes to advancing the state of PD diagnostics and improving patient care through innovative, multimodal approaches.
Abstract (translated)
帕金森病（PD）是一种进行性神经退行性疾病，主要影响运动功能，并可能导致其晚期阶段出现轻度认知障碍（MCI）和痴呆。据《日本时报》和帕金森基金会的报告，全球约有1000万患者，患病率为每千人中有1.8例。早期且准确地诊断PD对于改善患者的预后至关重要。尽管许多研究已经利用机器学习（ML）和深度学习（DL）技术进行PD识别，但现有的综述往往在范围上有所局限，常常仅关注单一的数据模态，并未能捕捉到多模态方法的潜力。 为了填补这些空白，本研究对多种数据模态下的帕金森病识别系统进行了全面回顾。这包括磁共振成像（MRI）、基于步态的姿态分析、步态感觉数据、手写分析、语音测试数据、脑电图（EEG）以及多模态融合技术等。根据来自领先科学数据库的347篇以上文献，本综述探讨了数据收集方法、设置、特征表示及系统性能的关键方面，并重点关注识别准确性和鲁棒性。 此次调查旨在为研究人员提供一个全面资源，为其开发下一代帕金森病识别系统的行动指南。通过利用多样化的数据模态和尖端的机器学习范式，这项工作有助于推进PD诊断技术的进步，并通过创新、多模态的方法改善患者的护理质量。
URL
https://arxiv.org/abs/2505.00525
PDF
-
Hashigo: A Next Generation Sketch Interactive System for Japanese Kanji
Abstract
Language students can increase their effectiveness in learning written Japanese by mastering the visual structure and written technique of Japanese kanji. Yet, existing kanji handwriting recognition systems do not assess the written technique sufficiently enough to discourage students from developing bad learning habits. In this paper, we describe our work on Hashigo, a kanji sketch interactive system which achieves human instructor-level critique and feedback on both the visual structure and written technique of students' sketched kanji. This type of automated critique and feedback allows students to target and correct specific deficiencies in their sketches that, if left untreated, are detrimental to effective long-term kanji learning.
Abstract (translated)
语言学习者可以通过掌握日语汉字的视觉结构和书写技巧来提高他们学习书面日语的效果。然而，现有的日语手写识别系统未能充分评估书写的技巧，不足以阻止学生养成不良的学习习惯。在本文中，我们介绍了我们在Hashigo项目上的工作，这是一个日语汉字草图互动系统，能够对学生的草图进行视觉结构和书写技巧方面的评判与反馈，其水平达到了人类教师的标准。这种自动化的评判和反馈可以让学生针对并纠正他们草图中的具体缺陷，这些缺陷如果不加以改正，在长期有效学习汉字的过程中将产生不利影响。
URL
https://arxiv.org/abs/2504.13940
PDF
-
Preserving Privacy Without Compromising Accuracy: Machine Unlearning for Handwritten Text Recognition
2025-04-11 15:21:12Lei Kang, Xuanshuo Fu, Lluis Gomez, Alicia Forn\'es, Ernest Valveny, Dimosthenis KaratzasAbstract
Handwritten Text Recognition (HTR) is essential for document analysis and digitization. However, handwritten data often contains user-identifiable information, such as unique handwriting styles and personal lexicon choices, which can compromise privacy and erode trust in AI services. Legislation like the ``right to be forgotten'' underscores the necessity for methods that can expunge sensitive information from trained models. Machine unlearning addresses this by selectively removing specific data from models without necessitating complete retraining. Yet, it frequently encounters a privacy-accuracy tradeoff, where safeguarding privacy leads to diminished model performance. In this paper, we introduce a novel two-stage unlearning strategy for a multi-head transformer-based HTR model, integrating pruning and random labeling. Our proposed method utilizes a writer classification head both as an indicator and a trigger for unlearning, while maintaining the efficacy of the recognition head. To our knowledge, this represents the first comprehensive exploration of machine unlearning within HTR tasks. We further employ Membership Inference Attacks (MIA) to evaluate the effectiveness of unlearning user-identifiable information. Extensive experiments demonstrate that our approach effectively preserves privacy while maintaining model accuracy, paving the way for new research directions in the document analysis community. Our code will be publicly available upon acceptance.
Abstract (translated)
手写文本识别（HTR）在文档分析和数字化中至关重要。然而，手写数据常常包含可辨识个人的信息，例如独特的书写风格和个人词汇选择，这可能危及隐私并削弱用户对AI服务的信任。类似于“被遗忘权”这样的立法强调了从训练模型中清除敏感信息的方法的必要性。机器卸载通过有选择地从模型中移除特定数据而不必重新进行全面训练来解决这个问题。然而，在保护隐私和保证准确性之间，经常存在一种折衷，即保护隐私会导致模型性能下降。 本文介绍了针对基于多头变换器的手写文本识别（HTR）模型的一种新的两阶段卸载策略，该策略结合了修剪和随机标签处理。我们提出的方法利用书写者分类头作为卸载的指标和触发器，同时保持识别头的有效性。据我们所知，这是首次全面探讨在HTR任务中机器卸载的研究。 此外，我们使用成员资格推理攻击（MIA）来评估删除可辨识用户信息的效果。广泛的实验表明，我们的方法能够有效地保护隐私的同时保持模型的准确性，为文档分析社区的新研究方向铺平了道路。我们的代码将在接受后公开提供。
URL
https://arxiv.org/abs/2504.08616
PDF
-
Enhancing Physical Human-Robot Interaction: Recognizing Digits via Intrinsic Robot Tactile Sensing
2025-03-31 19:22:01Teresa Sinico, Giovanni Boschetti, Pedro NetoAbstract
Physical human-robot interaction (pHRI) remains a key challenge for achieving intuitive and safe interaction with robots. Current advancements often rely on external tactile sensors as interface, which increase the complexity of robotic systems. In this study, we leverage the intrinsic tactile sensing capabilities of collaborative robots to recognize digits drawn by humans on an uninstrumented touchpad mounted to the robot's flange. We propose a dataset of robot joint torque signals along with corresponding end-effector (EEF) forces and moments, captured from the robot's integrated torque sensors in each joint, as users draw handwritten digits (0-9) on the touchpad. The pHRI-DIGI-TACT dataset was collected from different users to capture natural variations in handwriting. To enhance classification robustness, we developed a data augmentation technique to account for reversed and rotated digits inputs. A Bidirectional Long Short-Term Memory (Bi-LSTM) network, leveraging the spatiotemporal nature of the data, performs online digit classification with an overall accuracy of 94\% across various test scenarios, including those involving users who did not participate in training the system. This methodology is implemented on a real robot in a fruit delivery task, demonstrating its potential to assist individuals in everyday life. Dataset and video demonstrations are available at: this https URL.
Abstract (translated)
物理人机交互（pHRI）仍然是实现直观和安全的人机互动的关键挑战。当前的进展通常依赖于外部触觉传感器作为接口，这增加了机器人的复杂性。在这项研究中，我们利用协作机器人本身的内在触觉感应能力来识别人类在未配备任何仪器的安装于机械臂法兰上的触摸板上书写的数字（0-9）。我们提出了一组数据集，该数据集包括从每个关节集成扭矩传感器捕获的机器人的关节扭矩信号以及相应的末端执行器（EEF）力和力矩。该pHRI-DIGI-TACT数据集来自不同的用户，以捕捉手写风格中的自然变化。为了增强分类的鲁棒性，我们开发了一种数据增强技术来应对倒置和旋转数字输入的情况。 采用双向长短期记忆（Bi-LSTM）网络利用数据的空间时间和序列特性，在各种测试场景中实现了在线数字识别，整体准确率达到了94%，包括那些未参与训练系统用户的测试。该方法在水果配送任务上通过真实机器人进行了实施，展示了其在未来日常生活中帮助个人的潜力。 数据集和视频演示可以在以下网址找到：this https URL.
URL
https://arxiv.org/abs/2504.00167
PDF
-
The Cursive Transformer
2025-03-31 03:22:27Sam Greydanus, Zachary WimpeeAbstract
Transformers trained on tokenized text, audio, and images can generate high-quality autoregressive samples. But handwriting data, represented as sequences of pen coordinates, remains underexplored. We introduce a novel tokenization scheme that converts pen stroke offsets to polar coordinates, discretizes them into bins, and then turns them into sequences of tokens with which to train a standard GPT model. This allows us to capture complex stroke distributions without using any specialized architectures (eg. the mixture density network or the self-advancing ASCII attention head from Graves 2014). With just 3,500 handwritten words and a few simple data augmentations, we are able to train a model that can generate realistic cursive handwriting. Our approach is simpler and more performant than previous RNN-based methods.
Abstract (translated)
基于标记化的文本、音频和图像训练的Transformers能够生成高质量的自回归样本。然而，手写数据以笔迹坐标序列的形式表示时，则较少被探索。我们引入了一种新颖的标记化方案，该方案将笔划偏移转换为极坐标，并将其离散化为若干区间，然后将这些区间转化为用于训练标准GPT模型的令牌序列。这使我们在不使用任何专门架构（如Graves 2014年提出的混合密度网络或自我推进ASCII注意头）的情况下捕捉复杂的笔迹分布成为可能。仅通过3,500个手写单词以及一些简单的数据增强，我们就能训练出能够生成逼真的连贯书写的手写模型。我们的方法比以前基于RNN的方法更简单且性能更好。
URL
https://arxiv.org/abs/2504.00051
PDF
-
Zero-Shot Styled Text Image Generation, but Make It Autoregressive
2025-03-21 11:56:20Vittorio Pippi, Fabio Quattrini, Silvia Cascianelli, Alessio Tonioni, Rita CucchiaraAbstract
Styled Handwritten Text Generation (HTG) has recently received attention from the computer vision and document analysis communities, which have developed several solutions, either GAN- or diffusion-based, that achieved promising results. Nonetheless, these strategies fail to generalize to novel styles and have technical constraints, particularly in terms of maximum output length and training efficiency. To overcome these limitations, in this work, we propose a novel framework for text image generation, dubbed Emuru. Our approach leverages a powerful text image representation model (a variational autoencoder) combined with an autoregressive Transformer. Our approach enables the generation of styled text images conditioned on textual content and style examples, such as specific fonts or handwriting styles. We train our model solely on a diverse, synthetic dataset of English text rendered in over 100,000 typewritten and calligraphy fonts, which gives it the capability to reproduce unseen styles (both fonts and users' handwriting) in zero-shot. To the best of our knowledge, Emuru is the first autoregressive model for HTG, and the first designed specifically for generalization to novel styles. Moreover, our model generates images without background artifacts, which are easier to use for downstream applications. Extensive evaluation on both typewritten and handwritten, any-length text image generation scenarios demonstrates the effectiveness of our approach.
Abstract (translated)
近期，带有风格的手写文本生成（HTG）技术受到了计算机视觉和文档分析社区的广泛关注。这些社区已经开发出几种解决方案，无论是基于GAN还是扩散模型的方法，都取得了相当不错的成果。然而，这些策略在面对新型样式时无法实现泛化，并且存在诸如最大输出长度和技术限制等具体问题。 为了解决这些问题，在这项工作中我们提出了一种新的文本图像生成框架，命名为Emuru。我们的方法结合了一个强大的文本图像表示模型（一种变分自编码器）与一个自回归的Transformer。这种方法能够基于文字内容和风格示例来生成带有特定字体或手写样式的文本图像。我们仅使用包含超过10万种打字机和书法字体的合成多样化英语文本数据集训练我们的模型，这使得它能够在零样本情况下再现未见过的样式（包括字体和用户的手写）。据我们所知，Emuru是第一个用于HTG的自回归模型，并且也是首个专门设计以泛化到新风格的模型。此外，我们的模型生成的图像不包含背景伪影，更容易应用于下游任务。 在打字机文本及手写文本（任意长度）的图像生成场景中进行的广泛评估显示了我们方法的有效性。
URL
https://arxiv.org/abs/2503.17074
PDF
-
Benchmarking Large Language Models for Handwritten Text Recognition
2025-03-19 13:33:29Giorgia Crosilla, Lukas Klic, Giovanni ColavizzaAbstract
Traditional machine learning models for Handwritten Text Recognition (HTR) rely on supervised training, requiring extensive manual annotations, and often produce errors due to the separation between layout and text processing. In contrast, Multimodal Large Language Models (MLLMs) offer a general approach to recognizing diverse handwriting styles without the need for model-specific training. The study benchmarks various proprietary and open-source LLMs against Transkribus models, evaluating their performance on both modern and historical datasets written in English, French, German, and Italian. In addition, emphasis is placed on testing the models' ability to autonomously correct previously generated outputs. Findings indicate that proprietary models, especially Claude 3.5 Sonnet, outperform open-source alternatives in zero-shot settings. MLLMs achieve excellent results in recognizing modern handwriting and exhibit a preference for the English language due to their pre-training dataset composition. Comparisons with Transkribus show no consistent advantage for either approach. Moreover, LLMs demonstrate limited ability to autonomously correct errors in zero-shot transcriptions.
Abstract (translated)
传统的手写文本识别（HTR）模型依赖于监督训练，需要大量的手动注释，并且由于布局处理和文字处理的分离常常会产生错误。相比之下，多模态大型语言模型（MLLMs）提供了一种通用的方法来识别多样化的书写风格，而无需特定于模型的训练。这项研究对各种专有和开源的大规模语言模型与Transkribus模型进行了基准测试，并评估了它们在现代和历史数据集上的表现，这些数据集包含英文、法文、德文和意大利文的手写文本。此外，该研究还强调了测试模型自主纠正先前生成的输出的能力。 研究结果表明，在零样本设置中，专有模型（特别是Claude 3.5 Sonnet）优于开源替代品。MLLMs在识别现代手写方面表现出色，并且由于其预训练数据集的构成而对英语表现出偏好。与Transkribus相比，没有一种方法显示出持续的优势。此外，在零样本转录中，LLMs自主纠正错误的能力有限。 简而言之，这项研究展示了MLLMs在处理多样化书写风格和现代手写方面的优越性能，并揭示了这些模型在不同语言背景下存在的局限性以及它们纠正自身错误能力的不足之处。
URL
https://arxiv.org/abs/2503.15195
PDF
