Abstract
Handwritten Text Generation (HTG) conditioned on text and style is a challenging task due to the variability of inter-user characteristics and the unlimited combinations of characters that form new words unseen during training. Diffusion Models have recently shown promising results in HTG but still remain under-explored. We present DiffusionPen (DiffPen), a 5-shot style handwritten text generation approach based on Latent Diffusion Models. By utilizing a hybrid style extractor that combines metric learning and classification, our approach manages to capture both textual and stylistic characteristics of seen and unseen words and styles, generating realistic handwritten samples. Moreover, we explore several variation strategies of the data with multi-style mixtures and noisy embeddings, enhancing the robustness and diversity of the generated data. Extensive experiments using IAM offline handwriting database show that our method outperforms existing methods qualitatively and quantitatively, and its additional generated data can improve the performance of Handwriting Text Recognition (HTR) systems. The code is available at: this https URL.
Abstract (translated)
手写文本生成（HTG）根据文本和风格进行条件是一个具有挑战性的任务，因为用户特征的变异性以及训练过程中形成的未见过的字符组合的无限可能性。扩散模型最近在HTG上取得了良好的效果，但仍然有待进一步探索。我们提出了一种基于潜在扩散模型的5轮手写文本生成方法：DiffusionPen。通过结合度量学习和分类，我们的方法能够捕捉见到的和未见到的单词及其风格的文本和风格特征，生成真实的手写样本。此外，我们探讨了使用多风格混合和噪声嵌入来增强数据的数据变化策略，从而提高生成的数据的稳健性和多样性。使用IAM离线手写数据库进行的大量实验证明，我们的方法在质量和数量上优于现有方法，其额外生成的数据可以提高手写文本识别（HTR）系统的性能。代码可在此处访问：https:// this URL。
URL
https://arxiv.org/abs/2409.06065
