Handwriting
-
Synthesising Handwritten Music with GANs: A Comprehensive Evaluation of CycleWGAN, ProGAN, and DCGAN
2024-11-25 14:10:43Elona Shatri, Kalikidhar Palavala, George FazekasAbstract
The generation of handwritten music sheets is a crucial step toward enhancing Optical Music Recognition (OMR) systems, which rely on large and diverse datasets for optimal performance. However, handwritten music sheets, often found in archives, present challenges for digitisation due to their fragility, varied handwriting styles, and image quality. This paper addresses the data scarcity problem by applying Generative Adversarial Networks (GANs) to synthesise realistic handwritten music sheets. We provide a comprehensive evaluation of three GAN models - DCGAN, ProGAN, and CycleWGAN - comparing their ability to generate diverse and high-quality handwritten music images. The proposed CycleWGAN model, which enhances style transfer and training stability, significantly outperforms DCGAN and ProGAN in both qualitative and quantitative evaluations. CycleWGAN achieves superior performance, with an FID score of 41.87, an IS of 2.29, and a KID of 0.05, making it a promising solution for improving OMR systems.
Abstract (translated)
URL
https://arxiv.org/abs/2411.16405
PDF
-
Learning based Ge'ez character handwritten recognition
2024-11-20 14:22:15Hailemicael Lulseged Yimer, Hailegabriel Dereje Degefa, Marco Cristani, Federico CunicoAbstract
Ge'ez, an ancient Ethiopic script of cultural and historical significance, has been largely neglected in handwriting recognition research, hindering the digitization of valuable manuscripts. Our study addresses this gap by developing a state-of-the-art Ge'ez handwriting recognition system using Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks. Our approach uses a two-stage recognition process. First, a CNN is trained to recognize individual characters, which then acts as a feature extractor for an LSTM-based system for word recognition. Our dual-stage recognition approach achieves new top scores in Ge'ez handwriting recognition, outperforming eight state-of-the-art methods, which are SVTR, ASTER, and others as well as human performance, as measured in the HHD-Ethiopic dataset work. This research significantly advances the preservation and accessibility of Ge'ez cultural heritage, with implications for historical document digitization, educational tools, and cultural preservation. The code will be released upon acceptance.
Abstract (translated)
古埃塞俄比亚文字盖兹文（Ge'ez），作为一种具有文化和历史意义的古老书写系统，长期以来在手写识别研究中被忽视，这阻碍了珍贵手稿的数字化进程。我们的研究通过开发一个基于卷积神经网络（CNNs）和长短时记忆（LSTM）网络的最先进的盖兹文手写识别系统来解决这一问题。我们的方法采用两阶段识别过程：首先训练一个 CNN 来识别单个字符，然后将该 CNN 作为特征提取器用于基于 LSTM 的单词识别系统。这种双阶段识别方法在盖兹文手写识别中达到了新的最高分，超越了包括 SVTR、ASTER 在内的八种最先进的方法以及人类的表现，这一结果是在 HHD-Ethiopic 数据集上进行测量得出的。这项研究显著推进了盖兹文文化遗产的保护和访问性，在历史文档数字化、教育工具及文化保存方面具有深远影响。代码将在接受后发布。
URL
https://arxiv.org/abs/2411.13350
PDF
-
Towards Accessible Learning: Deep Learning-Based Potential Dysgraphia Detection and OCR for Potentially Dysgraphic Handwriting
2024-11-18 13:28:26Vydeki D, Divyansh Bhandari, Pranav Pratap Patil, Aarush Anand KulkarniAbstract
Dysgraphia is a learning disorder that affects handwriting abilities, making it challenging for children to write legibly and consistently. Early detection and monitoring are crucial for providing timely support and interventions. This study applies deep learning techniques to address the dual tasks of dysgraphia detection and optical character recognition (OCR) on handwriting samples from children with potential dysgraphic symptoms. Using a dataset of handwritten samples from Malaysian schoolchildren, we developed a custom Convolutional Neural Network (CNN) model, alongside VGG16 and ResNet50, to classify handwriting as dysgraphic or non-dysgraphic. The custom CNN model outperformed the pre-trained models, achieving a test accuracy of 91.8% with high precision, recall, and AUC, demonstrating its robustness in identifying dysgraphic handwriting features. Additionally, an OCR pipeline was created to segment and recognize individual characters in dysgraphic handwriting, achieving a character recognition accuracy of approximately 43.5%. This research highlights the potential of deep learning in supporting dysgraphia assessment, laying a foundation for tools that could assist educators and clinicians in identifying dysgraphia and tracking handwriting progress over time. The findings contribute to advancements in assistive technologies for learning disabilities, offering hope for more accessible and accurate diagnostic tools in educational and clinical settings.
Abstract (translated)
发育性书写障碍是一种影响书写能力的学习障碍，使得儿童难以清晰且一致地书写。早期检测和监控对于提供及时的支持和干预至关重要。本研究应用深度学习技术来解决识别发育性书写障碍及手写样本光学字符识别（OCR）的双重任务。我们使用了一组来自马来西亚学生的手写样本数据集，开发了一个自定义卷积神经网络（CNN）模型，并与VGG16和ResNet50进行了对比，以对手写体进行分类，判断其是否为发育性书写障碍。自定义CNN模型的表现优于预训练模型，在测试中达到了91.8%的准确率，具有高精确度、召回率和AUC值，展示了其在识别发育性书写特征方面的稳健性。此外，还建立了一个OCR管道来分割并识别发育性书写障碍手写体中的单个字符，实现了大约43.5%的字符识别准确性。本研究强调了深度学习在支持发育性书写障碍评估方面的潜力，为开发有助于教育者和临床医生识别发育性书写障碍及跟踪随时间推移的书写进步情况的工具奠定了基础。该研究成果促进了辅助技术的发展，为学习障碍提供了更加便捷且准确的诊断工具，在教育和临床环境中带来了希望。
URL
https://arxiv.org/abs/2411.13595
PDF
-
Towards Scalable Handwriting Communication via EEG Decoding and Latent Embedding Integration
2024-11-14 04:12:47Jun-Young Kim, Deok-Seon Kim, Seo-Hyun LeeAbstract
In recent years, brain-computer interfaces have made advances in decoding various motor-related tasks, including gesture recognition and movement classification, utilizing electroencephalogram (EEG) data. These developments are fundamental in exploring how neural signals can be interpreted to recognize specific physical actions. This study centers on a written alphabet classification task, where we aim to decode EEG signals associated with handwriting. To achieve this, we incorporate hand kinematics to guide the extraction of the consistent embeddings from high-dimensional neural recordings using auxiliary variables (CEBRA). These CEBRA embeddings, along with the EEG, are processed by a parallel convolutional neural network model that extracts features from both data sources simultaneously. The model classifies nine different handwritten characters, including symbols such as exclamation marks and commas, within the alphabet. We evaluate the model using a quantitative five-fold cross-validation approach and explore the structure of the embedding space through visualizations. Our approach achieves a classification accuracy of 91 % for the nine-class task, demonstrating the feasibility of fine-grained handwriting decoding from EEG.
Abstract (translated)
近年来，脑机接口在解码各种与运动相关的任务方面取得了进展，包括通过脑电图（EEG）数据进行手势识别和动作分类。这些发展对于探索如何解读神经信号以识别特定的物理动作具有基础性意义。本研究专注于手写字母分类任务，旨在解码与书写相关的大脑电信号。为此，我们整合了手部运动学知识来指导利用辅助变量（CEBRA）从高维神经记录中提取一致嵌入的过程。这些CEBRA嵌入和EEG数据由一个并行卷积神经网络模型处理，该模型同时从两个数据源中提取特征。模型对包括感叹号和逗点等符号在内的九种不同手写字符进行分类。我们使用定量的五折交叉验证方法评估了模型，并通过可视化探索了嵌入空间的结构。我们的方法在九类任务上达到了91%的分类准确率，这证明了从EEG信号中精细解码手写动作的可行性。
URL
https://arxiv.org/abs/2411.09170
PDF
-
Evaluation of handwriting kinematics and pressure for differential diagnosis of Parkinson's disease
2024-11-05 12:27:24Peter Drot\'ar, Ji\v{r}\'i Mekyska, Irena Rektorov\'a, Lucia Masarov\'a, Zden\v{e}k Sm\'ekal, Marcos Faundez-ZanuyAbstract
Objective: We present the PaHaW Parkinson's disease handwriting database, consisting of handwriting samples from Parkinson's disease (PD) patients and healthy controls. Our goal is to show that kinematic features and pressure features in handwriting can be used for the differential diagnosis of PD. Methods and Material: The database contains records from 37 PD patients and 38 healthy controls performing eight different handwriting tasks. The tasks include drawing an Archimedean spiral, repetitively writing orthographically simple syllables and words, and writing of a sentence. In addition to the conventional kinematic features related to the dynamics of handwriting, we investigated new pressure features based on the pressure exerted on the writing surface. To discriminate between PD patients and healthy subjects, three different classifiers were compared: K-nearest neighbors (K-NN), ensemble AdaBoost classifier, and support vector machines (SVM). Results: For predicting PD based on kinematic and pressure features of handwriting, the best performing model was SVM with classification accuracy of Pacc = 81.3% (sensitivity Psen = 87.4% and specificity of Pspe = 80.9%). When evaluated separately, pressure features proved to be relevant for PD diagnosis, yielding Pacc = 82.5% compared to Pacc = 75.4% using kinematic features. Conclusion: Experimental results showed that an analysis of kinematic and pressure features during handwriting can help assess subtle characteristics of handwriting and discriminate between PD patients and healthy controls.
Abstract (translated)
目标：我们提出了PaHaW帕金森病手写数据库，该数据库包含来自帕金森病（PD）患者和健康对照组的手写样本。我们的目标是展示手写的动力学特征和压力特征可用于帕金森病的鉴别诊断。 方法与材料：该数据库包括37名PD患者和38名健康对照者执行八种不同手写任务的记录。这些任务包括绘制阿基米德螺旋线，反复书写正字法简单的音节和单词，以及书写句子。除了常规的手写动力学特征外，我们还研究了基于对手写表面施加的压力的新压力特征。为了区分PD患者和健康受试者，比较了三种不同的分类器：K最近邻（K-NN）、集成AdaBoost分类器和支持向量机（SVM）。 结果：在根据手写的动力学和压力特征预测帕金森病方面，表现最好的模型是支持向量机（SVM），其分类准确率为Pacc = 81.3%（敏感性Psen = 87.4%，特异性Pspe = 80.9%）。单独评估时，压力特征对于PD诊断具有相关性，得出的准确率为Pacc = 82.5%，而使用动力学特征得到的准确率为Pacc = 75.4%。 结论：实验结果显示，在手写过程中对手写的动力学和压力特征进行分析可以帮助评估手写的细微特性，并区分帕金森病患者与健康对照组。
URL
https://arxiv.org/abs/2411.03044
PDF
-
State- and context-dependent robotic manipulation and grasping via uncertainty-aware imitation learning
2024-10-31 15:32:32Tim R. Winter, Ashok M. Sundaram, Werner Friedl, Maximo A. Roa, Freek Stulp, Jo\~ao Silv\'erioAbstract
Generating context-adaptive manipulation and grasping actions is a challenging problem in robotics. Classical planning and control algorithms tend to be inflexible with regard to parameterization by external variables such as object shapes. In contrast, Learning from Demonstration (LfD) approaches, due to their nature as function approximators, allow for introducing external variables to modulate policies in response to the environment. In this paper, we utilize this property by introducing an LfD approach to acquire context-dependent grasping and manipulation strategies. We treat the problem as a kernel-based function approximation, where the kernel inputs include generic context variables describing task-dependent parameters such as the object shape. We build on existing work on policy fusion with uncertainty quantification to propose a state-dependent approach that automatically returns to demonstrations, avoiding unpredictable behavior while smoothly adapting to context changes. The approach is evaluated against the LASA handwriting dataset and on a real 7-DoF robot in two scenarios: adaptation to slippage while grasping and manipulating a deformable food item.
Abstract (translated)
生成适应环境的操纵和抓取动作是机器人技术中的一个挑战性问题。传统的规划和控制算法在对外部变量（如物体形状）进行参数化时往往缺乏灵活性。相比之下，由于演示学习（LfD）方法本质上是函数近似器，它们允许引入外部变量来调节策略以应对环境变化。在这篇论文中，我们利用这一特性，采用一种演示学习方法来获取依赖于上下文的抓取和操纵策略。我们将问题视为基于核的函数逼近，其中核输入包括描述任务相关参数（如物体形状）的一般上下文变量。我们在现有政策融合及不确定性量化工作的基础上提出了一种状态相关的方案，该方案能自动返回演示，避免不可预测的行为，并平滑地适应环境变化。此方法在LASA手写数据集上进行了评估，并在一个真实的7自由度机器人上两种场景下得到了验证：抓取和操纵可变形食品项目时对打滑的适应性。
URL
https://arxiv.org/abs/2410.24035
PDF
-
Handwriting Recognition in Historical Documents with Multimodal LLM
2024-10-31 15:32:14Lucian LiAbstract
There is an immense quantity of historical and cultural documentation that exists only as handwritten manuscripts. At the same time, performing OCR across scripts and different handwriting styles has proven to be an enormously difficult problem relative to the process of digitizing print. While recent Transformer based models have achieved relatively strong performance, they rely heavily on manually transcribed training data and have difficulty generalizing across writers. Multimodal LLM, such as GPT-4v and Gemini, have demonstrated effectiveness in performing OCR and computer vision tasks with few shot prompting. In this paper, I evaluate the accuracy of handwritten document transcriptions generated by Gemini against the current state of the art Transformer based methods. Keywords: Optical Character Recognition, Multimodal Language Models, Cultural Preservation, Mass digitization, Handwriting Recognitio
Abstract (translated)
有大量的历史和文化文档仅以手稿形式存在。同时，跨不同书写脚本和风格进行OCR（光学字符识别）证明比数字化印刷品的过程要困难得多。尽管最近基于Transformer的模型已经实现了相对较强的表现力，但它们高度依赖于手动转录的训练数据，并且在面对不同的写作者时难以泛化。多模态语言模型如GPT-4v和Gemini已经在使用少量样本提示的情况下展示了执行OCR和计算机视觉任务的有效性。在这篇论文中，我评估了由Gemini生成的手稿文档转录准确度与当前最先进的基于Transformer的方法之间的对比情况。关键词：光学字符识别，多模态语言模型，文化保存，大规模数字化，手写识别
URL
https://arxiv.org/abs/2410.24034
PDF
-
Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
2024-10-24 17:59:21David Ortiz-Perez, Manuel Benavent-Lledo, Jose Garcia-Rodriguez, David Tom\'as, M. Flores Vizcaya-MorenoAbstract
Cognitive decline is a natural part of aging, often resulting in reduced cognitive abilities. In some cases, however, this decline is more pronounced, typically due to disorders such as Alzheimer's disease. Early detection of anomalous cognitive decline is crucial, as it can facilitate timely professional intervention. While medical data can help in this detection, it often involves invasive procedures. An alternative approach is to employ non-intrusive techniques such as speech or handwriting analysis, which do not necessarily affect daily activities. This survey reviews the most relevant methodologies that use deep learning techniques to automate the cognitive decline estimation task, including audio, text, and visual processing. We discuss the key features and advantages of each modality and methodology, including state-of-the-art approaches like Transformer architecture and foundation models. In addition, we present works that integrate different modalities to develop multimodal models. We also highlight the most significant datasets and the quantitative results from studies using these resources. From this review, several conclusions emerge. In most cases, the textual modality achieves the best results and is the most relevant for detecting cognitive decline. Moreover, combining various approaches from individual modalities into a multimodal model consistently enhances performance across nearly all scenarios.
Abstract (translated)
认知衰退是老化过程中的自然现象，通常会导致认知能力的下降。然而，在某些情况下，这种衰退更加明显，通常是由于阿尔茨海默病等疾病所致。早期检测异常的认知衰退至关重要，因为它可以促进及时的专业干预。虽然医学数据有助于此类检测，但往往涉及侵入性程序。一种替代方法是使用非侵扰性的技术，如语音或笔迹分析，这些技术通常不会影响日常活动。本综述回顾了利用深度学习技术自动化认知衰退估计任务的最相关方法，包括音频、文本和视觉处理。我们讨论了每种模态及其方法的关键特征与优势，包括最先进的方法如Transformer架构和基础模型。此外，我们还介绍了整合不同模态以开发多模态模型的工作成果。我们也强调了一些最重要的数据集以及利用这些资源的研究的定量结果。从这一综述中，我们可以得出几个结论。在大多数情况下，文本模态获得了最佳效果，并且对于检测认知衰退最为相关。此外，在几乎所有场景下，将来自单个模态的各种方法结合到一个多模态模型中可以持续提升性能。
URL
https://arxiv.org/abs/2410.18972
PDF
-
Explainable AI in Handwriting Detection for Dyslexia Using Transfer Learning
2024-10-18 11:14:54Mahmoud Robaa, Mazen Balat, Rewaa Awaad, Esraa Omar, Salah A. AlyAbstract
Dyslexia is one of the most common learning disorders, often characterized by distinct features in handwriting. Early detection is essential for effective intervention. In this paper, we propose an explainable AI (XAI) framework for dyslexia detection through handwriting analysis, utilizing transfer learning and transformer-based models. Our approach surpasses state-of-the-art methods, achieving a test accuracy of 0.9958, while ensuring model interpretability through Grad-CAM visualizations that highlight the critical handwriting features influencing model decisions. The main contributions of this work include the integration of XAI for enhanced interpretability, adaptation to diverse languages and writing systems, and demonstration of the method's global applicability. This framework not only improves diagnostic accuracy but also fosters trust and understanding among educators, clinicians, and parents, supporting earlier diagnoses and the development of personalized educational strategies.
Abstract (translated)
阅读障碍是常见的学习障碍之一，通常通过手写特征表现出独特的特点。早期检测对于有效干预至关重要。本文提出了一种通过手写分析来检测阅读障碍的可解释人工智能（XAI）框架，该框架利用了迁移学习和基于变压器的模型。我们的方法超越了现有最佳方法，在测试中达到了0.9958的准确率，并且通过Grad-CAM可视化确保了模型的可解释性，突出了影响模型决策的关键手写特征。这项工作的主要贡献包括：为提高可解释性而整合XAI、适应多种语言和书写系统以及展示了该方法的全球适用性。此框架不仅提高了诊断准确性，而且促进了教育工作者、临床医生及家长之间的信任与理解，支持更早地进行诊断并开发个性化的教育策略。
URL
https://arxiv.org/abs/2410.19821
PDF
-
Hybrid Transformer for Early Alzheimer's Detection: Integration of Handwriting-Based 2D Images and 1D Signal Features
2024-10-14 14:26:52Changqing Gong, Huafeng Qin, Moun\^im A. El-YacoubiAbstract
Alzheimer's Disease (AD) is a prevalent neurodegenerative condition where early detection is vital. Handwriting, often affected early in AD, offers a non-invasive and cost-effective way to capture subtle motor changes. State-of-the-art research on handwriting, mostly online, based AD detection has predominantly relied on manually extracted features, fed as input to shallow machine learning models. Some recent works have proposed deep learning (DL)-based models, either 1D-CNN or 2D-CNN architectures, with performance comparing favorably to handcrafted schemes. These approaches, however, overlook the intrinsic relationship between the 2D spatial patterns of handwriting strokes and their 1D dynamic characteristics, thus limiting their capacity to capture the multimodal nature of handwriting data. Moreover, the application of Transformer models remains basically unexplored. To address these limitations, we propose a novel approach for AD detection, consisting of a learnable multimodal hybrid attention model that integrates simultaneously 2D handwriting images with 1D dynamic handwriting signals. Our model leverages a gated mechanism to combine similarity and difference attention, blending the two modalities and learning robust features by incorporating information at different scales. Our model achieved state-of-the-art performance on the DARWIN dataset, with an F1-score of 90.32\% and accuracy of 90.91\% in Task 8 ('L' writing), surpassing the previous best by 4.61% and 6.06% respectively.
Abstract (translated)
URL
https://arxiv.org/abs/2410.10547
PDF
-
MIRAGE: Multimodal Identification and Recognition of Annotations in Indian General Prescriptions
2024-10-13 05:19:09Tavish Mankash, V. S. Chaithanya Kota, Anish De, Praveen Prakash, Kshitij JadhavAbstract
Hospitals generate thousands of handwritten prescriptions, a practice that remains prevalent despite the availability of Electronic Medical Records (EMR). This method of record-keeping hinders the examination of long-term medication effects, impedes statistical analysis, and makes the retrieval of records challenging. Handwritten prescriptions pose a unique challenge, requiring specialized data for training models to recognize medications and their patterns of recommendation. While current handwriting recognition approaches typically employ 2-D LSTMs, recent studies have explored the use of Large Language Models (LLMs) for Optical Character Recognition (OCR). Building on this approach, we focus on extracting medication names from medical records. Our methodology MIRAGE (Multimodal Identification and Recognition of Annotations in indian GEneral prescriptions) involves fine-tuning the LLaVA 1.6 and Idefics2 models. Our research utilizes a dataset provided by Medyug Technology, consisting of 743,118 fully annotated high-resolution simulated medical records from 1,133 doctors across India. We demonstrate that our methodology exhibits 82% accuracy in medication name and dosage extraction. We provide a detailed account of our research methodology and results, notes about HWR with Multimodal LLMs, and release a small dataset of 100 medical records with labels.
Abstract (translated)
URL
https://arxiv.org/abs/2410.09729
PDF
-
Decoupling Layout from Glyph in Online Chinese Handwriting Generation
2024-10-03 08:46:17Ren-Min Si, Yan-Ming Zhang, Yi ChenAbstract
Text plays a crucial role in the transmission of human civilization, and teaching machines to generate online handwritten text in various styles presents an interesting and significant challenge. However, most prior work has concentrated on generating individual Chinese fonts, leaving {complete text line generation largely unexplored}. In this paper, we identify that text lines can naturally be divided into two components: layout and glyphs. Based on this division, we designed a text line layout generator coupled with a diffusion-based stylized font synthesizer to address this challenge hierarchically. More concretely, the layout generator performs in-context-like learning based on the text content and the provided style references to generate positions for each glyph autoregressively. Meanwhile, the font synthesizer which consists of a character embedding dictionary, a multi-scale calligraphy style encoder, and a 1D U-Net based diffusion denoiser will generate each font on its position while imitating the calligraphy style extracted from the given style references. Qualitative and quantitative experiments on the CASIA-OLHWDB demonstrate that our method is capable of generating structurally correct and indistinguishable imitation samples.
Abstract (translated)
文本在人类文明的传播中扮演着关键角色，并且教机器生成各种风格的在线手写文本提出了一个有趣且具有挑战性的问题。然而，大多数先前的研究都集中在生成单个中文字体上， leaving {完整的文本行生成大多没有被探索}。在本文中，我们发现文本行可以自然地分为两个部分：布局和字符。基于这一划分，我们设计了一个文本行布局生成器与扩散为基础的 stylized 字体合成器来解决这个问题。具体来说，布局生成器根据文本内容和提供的风格参考进行上下文类似的学习，以生成每个字符的自动位置。同时，由字符嵌入字典、多尺度书法风格编码器和基于 U-Net 的扩散去噪器组成的字体合成器将在其位置上模仿从给定风格参考中提取的书法风格。在 CASIA-OLHWDB 等数据集上进行的定性和定量实验证明，我们的方法能够生成结构正确且难以区分模仿样本。
URL
https://arxiv.org/abs/2410.02309
PDF
-
HATFormer: Historic Handwritten Arabic Text Recognition with Transformers
2024-10-03 03:43:29Adrian Chan, Anupam Mijar, Mehreen Saeed, Chau-Wai Wong, Akram KhaterAbstract
Arabic handwritten text recognition (HTR) is challenging, especially for historical texts, due to diverse writing styles and the intrinsic features of Arabic script. Additionally, Arabic handwriting datasets are smaller compared to English ones, making it difficult to train generalizable Arabic HTR models. To address these challenges, we propose HATFormer, a transformer-based encoder-decoder architecture that builds on a state-of-the-art English HTR model. By leveraging the transformer's attention mechanism, HATFormer captures spatial contextual information to address the intrinsic challenges of Arabic script through differentiating cursive characters, decomposing visual representations, and identifying diacritics. Our customization to historical handwritten Arabic includes an image processor for effective ViT information preprocessing, a text tokenizer for compact Arabic text representation, and a training pipeline that accounts for a limited amount of historic Arabic handwriting data. HATFormer achieves a character error rate (CER) of 8.6% on the largest public historical handwritten Arabic dataset, with a 51% improvement over the best baseline in the literature. HATFormer also attains a comparable CER of 4.2% on the largest private non-historical dataset. Our work demonstrates the feasibility of adapting an English HTR method to a low-resource language with complex, language-specific challenges, contributing to advancements in document digitization, information retrieval, and cultural preservation.
Abstract (translated)
阿拉伯手写文本识别（HTR）具有挑战性，尤其是在历史文本中，因为阿拉伯文本具有多样性的书写风格和阿拉伯文字本特征。此外，阿拉伯手写数据集比英语数据集要小，这使得训练具有泛化能力的阿拉伯HTR模型变得困难。为了应对这些挑战，我们提出了HATFormer，一种基于最先进的英语HTR模型的Transformer编码器-解码器架构。通过利用Transformer的注意力机制，HATFormer捕捉到阿拉伯文本的空间上下文信息，通过区分手写字符、分解视觉表示和识别变体来解决阿拉伯文字本固有的挑战。我们对历史手写阿拉伯的定制包括一个有效的ViT信息预处理图像处理器、一个紧凑的阿拉伯文本词条izer和一个考虑有限历史阿拉伯手写数据训练工作流的训练管道。HATFormer在最大的公共历史手写阿拉伯数据集上的字符错误率（CER）为8.6%，在文献中的最佳基线上的性能提高了51%。HATFormer还在最大的私人非历史数据集上获得了与文献中类似且可比的CER，为将英语HTR方法应用于具有复杂、语言特定挑战的低资源语言奠定了基础，有助于促进文档数字化、信息检索和文化遗产保护的发展。
URL
https://arxiv.org/abs/2410.02179
PDF
-
Learning from Demonstration with Implicit Nonlinear Dynamics Models
2024-09-27 14:12:49Peter David Fagan, Subramanian RamamoorthyAbstract
Learning from Demonstration (LfD) is a useful paradigm for training policies that solve tasks involving complex motions. In practice, the successful application of LfD requires overcoming error accumulation during policy execution, i.e. the problem of drift due to errors compounding over time and the consequent out-of-distribution behaviours. Existing works seek to address this problem through scaling data collection, correcting policy errors with a human-in-the-loop, temporally ensembling policy predictions or through learning the parameters of a dynamical system model. In this work, we propose and validate an alternative approach to overcoming this issue. Inspired by reservoir computing, we develop a novel neural network layer that includes a fixed nonlinear dynamical system with tunable dynamical properties. We validate the efficacy of our neural network layer on the task of reproducing human handwriting motions using the LASA Human Handwriting Dataset. Through empirical experiments we demonstrate that incorporating our layer into existing neural network architectures addresses the issue of compounding errors in LfD. Furthermore, we perform a comparative evaluation against existing approaches including a temporal ensemble of policy predictions and an Echo State Networks (ESNs) implementation. We find that our approach yields greater policy precision and robustness on the handwriting task while also generalising to multiple dynamics regimes and maintaining competitive latency scores.
Abstract (translated)
学习演示（LfD）是一种有效的解决涉及复杂动作任务的策略训练范式。在实践中，成功应用LfD需要通过在策略执行过程中克服误差累积问题，即误差随时间累积导致的漂移问题，以及随之而来的离散行为。现有作品试图通过缩放数据收集、通过人机交互来纠正策略错误、通过学习动态系统模型的参数来解决这个问题。在这篇工作中，我们提出了并验证了一种克服这个问题的替代方法。受到水库计算的启发，我们开发了一个新神经网络层，包括一个固定非线性动力学系统，具有可调的动态特性。我们通过使用LASA人类手写字符数据集来验证我们神经网络层的有效性。通过实验我们证明了将我们的层应用于现有的神经网络架构可以解决LfD中累积误差的問題。此外，我们还对包括基于策略预测的时间聚类和Echo State Networks（ESNs）实现的现有方法进行了比较评估。我们发现，与其他方法相比，我们的方法在手写字符任务中具有更高的策略精度、鲁棒性和扩展性，同时保持竞争延迟分数。
URL
https://arxiv.org/abs/2409.18768
PDF
-
DiffusionPen: Towards Controlling the Style of Handwritten Text Generation
2024-09-09 20:58:25Konstantina Nikolaidou, George Retsinas, Giorgos Sfikas, Marcus LiwickiAbstract
Handwritten Text Generation (HTG) conditioned on text and style is a challenging task due to the variability of inter-user characteristics and the unlimited combinations of characters that form new words unseen during training. Diffusion Models have recently shown promising results in HTG but still remain under-explored. We present DiffusionPen (DiffPen), a 5-shot style handwritten text generation approach based on Latent Diffusion Models. By utilizing a hybrid style extractor that combines metric learning and classification, our approach manages to capture both textual and stylistic characteristics of seen and unseen words and styles, generating realistic handwritten samples. Moreover, we explore several variation strategies of the data with multi-style mixtures and noisy embeddings, enhancing the robustness and diversity of the generated data. Extensive experiments using IAM offline handwriting database show that our method outperforms existing methods qualitatively and quantitatively, and its additional generated data can improve the performance of Handwriting Text Recognition (HTR) systems. The code is available at: this https URL.
Abstract (translated)
手写文本生成（HTG）根据文本和风格进行条件是一个具有挑战性的任务，因为用户特征的变异性以及训练过程中形成的未见过的字符组合的无限可能性。扩散模型最近在HTG上取得了良好的效果，但仍然有待进一步探索。我们提出了一种基于潜在扩散模型的5轮手写文本生成方法：DiffusionPen。通过结合度量学习和分类，我们的方法能够捕捉见到的和未见到的单词及其风格的文本和风格特征，生成真实的手写样本。此外，我们探讨了使用多风格混合和噪声嵌入来增强数据的数据变化策略，从而提高生成的数据的稳健性和多样性。使用IAM离线手写数据库进行的大量实验证明，我们的方法在质量和数量上优于现有方法，其额外生成的数据可以提高手写文本识别（HTR）系统的性能。代码可在此处访问：https:// this URL。
URL
https://arxiv.org/abs/2409.06065
PDF
-
Boosting CNN-based Handwriting Recognition Systems with Learnable Relaxation Labeling
2024-09-09 15:12:28Sara Ferro, Alessandro Torcinovich, Arianna Traviglia, Marcello PelilloAbstract
The primary challenge for handwriting recognition systems lies in managing long-range contextual dependencies, an issue that traditional models often struggle with. To mitigate it, attention mechanisms have recently been employed to enhance context-aware labelling, thereby achieving state-of-the-art performance. In the field of pattern recognition and image analysis, however, the use of contextual information in labelling problems has a long history and goes back at least to the early 1970's. Among the various approaches developed in those years, Relaxation Labelling (RL) processes have played a prominent role and have been the method of choice in the field for more than a decade. Contrary to recent transformer-based architectures, RL processes offer a principled approach to the use of contextual constraints, having a solid theoretic foundation grounded on variational inequality and game theory, as well as effective algorithms with convergence guarantees. In this paper, we propose a novel approach to handwriting recognition that integrates the strengths of two distinct methodologies. In particular, we propose integrating (trainable) RL processes with various well-established neural architectures and we introduce a sparsification technique that accelerates the convergence of the algorithm and enhances the overall system's performance. Experiments over several benchmark datasets show that RL processes can improve the generalisation ability, even surpassing in some cases transformer-based architectures.
Abstract (translated)
手写字符识别系统的主要挑战在于处理长距离上下文依赖关系，这是传统模型通常难以应对的问题。为解决这个问题，近年来采用了一种注意力机制来增强上下文敏感的标注，从而实现了最先进的性能。然而，在模式识别和图像分析领域，使用上下文信息进行标注问题已经有着悠久的历史，并且至少可以追溯到20世纪70年代初。在那几年里开发的各种方法中， relaxation标记（RL）过程发挥了重要作用，并且已经成为该领域的首选方法超过十年。与最近基于Transformer的架构不同，RL过程具有基于变分不等式和博弈论的严谨的理论基础，以及具有收敛保证的有效算法。在本文中，我们提出了一种将两个不同的方法论相结合的新型手写字符识别方法。特别地，我们提出将可训练的RL过程与各种已经确立的神经网络架构集成，并引入了一种稀疏化技术，加速算法的收敛，提高整体系统的性能。在多个基准数据集上的实验结果表明，RL过程可以提高泛化能力，甚至在某些情况下超过了基于Transformer的架构。
URL
https://arxiv.org/abs/2409.05699
PDF
-
One-Shot Diffusion Mimicker for Handwritten Text Generation
2024-09-06 03:10:59Gang Dai, Yifan Zhang, Quhui Ke, Qiangya Guo, Shuangping HuangAbstract
Existing handwritten text generation methods often require more than ten handwriting samples as style references. However, in practical applications, users tend to prefer a handwriting generation model that operates with just a single reference sample for its convenience and efficiency. This approach, known as "one-shot generation", significantly simplifies the process but poses a significant challenge due to the difficulty of accurately capturing a writer's style from a single sample, especially when extracting fine details from the characters' edges amidst sparse foreground and undesired background noise. To address this problem, we propose a One-shot Diffusion Mimicker (One-DM) to generate handwritten text that can mimic any calligraphic style with only one reference sample. Inspired by the fact that high-frequency information of the individual sample often contains distinct style patterns (e.g., character slant and letter joining), we develop a novel style-enhanced module to improve the style extraction by incorporating high-frequency components from a single sample. We then fuse the style features with the text content as a merged condition for guiding the diffusion model to produce high-quality handwritten text images. Extensive experiments demonstrate that our method can successfully generate handwriting scripts with just one sample reference in multiple languages, even outperforming previous methods using over ten samples. Our source code is available at this https URL.
Abstract (translated)
现有的手写文本生成方法通常需要超过10个手写样本作为样式参考。然而，在实际应用中，用户倾向于更喜欢只需一个手写样本操作的单样本生成模型，因为这样既方便又高效。这种方法被称为“一次生成”，显著简化了工艺流程，但由于准确捕捉一个写者的风格从单个样本的难度较大，尤其是在从字符边缘提取细小细节的过程中，尤其是在稀疏的前景和不需要的背景噪音中，这个问题仍然存在。为了应对这个问题，我们提出了一个一次生成扩散模仿者（One-DM）来生成具有任何楷体风格的只有一单个样本的手写文本。受到单个样本中高频信息通常包含不同的风格模式（例如，字符倾斜和字母连接）的影响，我们开发了一个新的样式增强模块，通过将高频组件从单个样本中集成来提高样式提取。然后将样式特征与文本内容作为合并条件，引导扩散模型产生高质量的手写文本图像。大量实验证明，我们的方法可以在多种语言中成功生成只需一个样本参考的手写文本，甚至超过了使用超过10个样本的前人方法。我们的源代码可在此处访问：https://url.cn/
URL
https://arxiv.org/abs/2409.04004
PDF
-
Rethinking HTG Evaluation: Bridging Generation and Recognition
2024-09-04 13:15:10Konstantina Nikolaidou, George Retsinas, Giorgos Sfikas, Marcus LiwickiAbstract
The evaluation of generative models for natural image tasks has been extensively studied. Similar protocols and metrics are used in cases with unique particularities, such as Handwriting Generation, even if they might not be completely appropriate. In this work, we introduce three measures tailored for HTG evaluation, $ \text{HTG}_{\text{HTR}} $, $ \text{HTG}_{\text{style}} $, and $ \text{HTG}_{\text{OOV}} $, and argue that they are more expedient to evaluate the quality of generated handwritten images. The metrics rely on the recognition error/accuracy of Handwriting Text Recognition and Writer Identification models and emphasize writing style, textual content, and diversity as the main aspects that adhere to the content of handwritten images. We conduct comprehensive experiments on the IAM handwriting database, showcasing that widely used metrics such as FID fail to properly quantify the diversity and the practical utility of generated handwriting samples. Our findings show that our metrics are richer in information and underscore the necessity of standardized evaluation protocols in HTG. The proposed metrics provide a more robust and informative protocol for assessing HTG quality, contributing to improved performance in HTR. Code for the evaluation protocol is available at: this https URL.
Abstract (translated)
自然图像任务的生成模型评估已经得到了广泛研究。在具有独特特点的情况下，如手写字符生成，即使它们可能不完全合适，也会使用类似的协议和指标。在这项工作中，我们引入了三个专门为HTG评估而设计的度量标准：$ \text{HTG}_{\text{HTR}} $，$ \text{HTG}_{\text{style}} $和$ \text{HTG}_{\text{OOV}} $，并认为它们更有效地评估生成手写字符图片的质量。这些指标基于手写字符识别错误/准确性和写者识别模型的识别误差/准确度，并强调书写风格，文本内容，多样性为主要方面，符合手写字符图片的内容。我们在IAM手写数据库上进行全面的实验，表明广泛使用的指标如FID无法正确量化生成手写字符样本的多样性和实际应用价值。我们的研究结果表明，我们的指标在信息量上更加丰富，进一步强调了在HTG中使用标准化评估协议的必要性。所提出的指标为评估HTG质量提供了一个更健壮和有益的协议，有助于提高HTR的性能。评估协议的代码可在此处下载：https:// this URL。
URL
https://arxiv.org/abs/2409.02683
PDF
-
BMI Prediction from Handwritten English Characters Using a Convolutional Neural Network
2024-09-04 10:06:42N. T. Diba, N. Akter, S. A. H. Chowdhury, J. E. GitiAbstract
A person's Body Mass Index, or BMI, is the most widely used parameter for assessing their health. BMI is a crucial predictor of potential diseases that may arise at higher body fat levels because it is correlated with body fat. Conversely, a community's or an individual's nutritional status can be determined using the BMI. Although deep learning models are used in several studies to estimate BMI from face photos and other data, no previous research established a clear connection between deep learning techniques for handwriting analysis and BMI prediction. This article addresses this research gap with a deep learning approach to estimating BMI from handwritten characters by developing a convolutional neural network (CNN). A dataset containing samples from 48 people in lowercase English scripts is successfully captured for the BMI prediction task. The proposed CNN-based approach reports a commendable accuracy of 99.92%. Performance comparison with other popular CNN architectures reveals that AlexNet and InceptionV3 achieve the second and third-best performance, with the accuracy of 99.69% and 99.53%, respectively.
Abstract (translated)
一个人的身体质量指数（BMI）是对其健康状况最广泛使用的参数。 BMI 与身体脂肪水平相关，是预测在较高身体脂肪水平可能出现的潜在疾病的關鍵預測因素。相反，一个社区或个人的營養狀況可以使用 BMI 來確定。雖然在多個研究中，使用面照片和其他數據來估計 BMI 的深度學習模型，但之前的研究沒有建立手寫字分析中深度學習技術與 BMI 預測之間的明確聯繫。本文通過使用深度學習方法估計手寫字中的 BMI，來填补這個研究缺口。開發了一個卷積神經網絡（CNN），通過估計手寫字中的 BMI，成功捕捉了包含48個人的小寫英文字母數據集的 BMI 預測任務數據集。與其他流行的 CNN 架構相比，基於 CNN 的方法報告了 99.92% 的優秀準確度。性能比較顯示，AlexNet 和 InceptionV3 實現了第二和第三好的性能，分別達到 99.69% 和 99.53%。
URL
https://arxiv.org/abs/2409.02584
PDF
-
Zero-Shot Paragraph-level Handwriting Imitation with Latent Diffusion Models
2024-09-01 17:33:31Martin Mayr, Marcel Dreier, Florian Kordon, Mathias Seuret, Jochen Z\"ollner, Fei Wu, Andreas Maier, Vincent ChristleinAbstract
The imitation of cursive handwriting is mainly limited to generating handwritten words or lines. Multiple synthetic outputs must be stitched together to create paragraphs or whole pages, whereby consistency and layout information are lost. To close this gap, we propose a method for imitating handwriting at the paragraph level that also works for unseen writing styles. Therefore, we introduce a modified latent diffusion model that enriches the encoder-decoder mechanism with specialized loss functions that explicitly preserve the style and content. We enhance the attention mechanism of the diffusion model with adaptive 2D positional encoding and the conditioning mechanism to work with two modalities simultaneously: a style image and the target text. This significantly improves the realism of the generated handwriting. Our approach sets a new benchmark in our comprehensive evaluation. It outperforms all existing imitation methods at both line and paragraph levels, considering combined style and content preservation.
Abstract (translated)
模仿草书手迹的主要限制在于只能生成手写单词或行。多个合成输出必须拼接在一起才能创建段落或整个页面，从而丢失了 consistency 和 layout 信息。为了填补这个空白，我们提出了一个模仿草书手迹的方法，该方法对于未见过的书写风格也有效。因此，我们引入了一个修改的潜在扩散模型，它通过专门损失函数丰富编码器-解码器机制，并保留风格和内容。我们通过自适应 2D 位置编码和条件机制来增强扩散模型的注意机制，以同时处理样式图像和目标文本。这使得生成的手写内容更真实。在我们的全面评估中，我们的方法为一个新的基准。它超越了所有现有的模仿方法，既考虑了风格，也考虑了内容保留。
URL
https://arxiv.org/abs/2409.00786
PDF
