Abstract
The generation of images of realistic looking, readable handwritten text is a challenging task which is referred to as handwritten text generation (HTG). Given a string and examples from a writer, the goal is to synthesize an image depicting the correctly spelled word in handwriting with the calligraphic style of the desired writer. An important application of HTG is the generation of training images in order to adapt downstream models for new data sets. With their success in natural image generation, diffusion models (DMs) have become the state-of-the-art approach in HTG. In this work, we present an extension of a latent DM for HTG to enable generation of writing styles not seen during training by learning style conditioning with a masked auto encoder. Our proposed content encoder allows for different ways of conditioning the DM on textual and calligraphic features. Additionally, we employ classifier-free guidance and explore the influence on the quality of the generated training images. For adapting the model to a new unlabeled data set, we propose a semi-supervised training scheme. We evaluate our approach on the IAM-database and use the RIMES-database to examine the generation of data not seen during training achieving improvements in this particularly promising application of DMs for HTG.
Abstract (translated)
生成看起来逼真且可读的手写文本图像是一个具有挑战性的任务，被称为手写文本生成（HTG）。给定一段字符串和某个书写者的样本，目标是合成一张图片，该图片以所期望的书写者风格展示正确拼写的单词。HTG的一个重要应用是在新数据集上训练下游模型时生成训练图像。由于在自然图像生成方面的成功，扩散模型（DMs）已成为HTG领域的领先方法。在此工作中，我们提出了一种扩展的潜在DM用于HTG，通过学习带有掩码自编码器的风格条件来实现生成未见过的书写风格。我们的内容编码器允许以不同的方式对DM进行文本和书法特征上的条件设置。此外，我们采用无分类器指导，并探讨其对生成训练图像质量的影响。为了将模型适应新的未标记数据集，我们提出了一种半监督训练方案。我们在IAM数据库上评估了我们的方法，并使用RIMES数据库来检查生成未见过的数据，从而在DMs用于HTG的这一特别有前景的应用中取得了改进。
URL
https://arxiv.org/abs/2412.15853
