Abstract
Styled Handwritten Text Generation (HTG) has recently received attention from the computer vision and document analysis communities, which have developed several solutions, either GAN- or diffusion-based, that achieved promising results. Nonetheless, these strategies fail to generalize to novel styles and have technical constraints, particularly in terms of maximum output length and training efficiency. To overcome these limitations, in this work, we propose a novel framework for text image generation, dubbed Emuru. Our approach leverages a powerful text image representation model (a variational autoencoder) combined with an autoregressive Transformer. Our approach enables the generation of styled text images conditioned on textual content and style examples, such as specific fonts or handwriting styles. We train our model solely on a diverse, synthetic dataset of English text rendered in over 100,000 typewritten and calligraphy fonts, which gives it the capability to reproduce unseen styles (both fonts and users' handwriting) in zero-shot. To the best of our knowledge, Emuru is the first autoregressive model for HTG, and the first designed specifically for generalization to novel styles. Moreover, our model generates images without background artifacts, which are easier to use for downstream applications. Extensive evaluation on both typewritten and handwritten, any-length text image generation scenarios demonstrates the effectiveness of our approach.
Abstract (translated)
近期，带有风格的手写文本生成（HTG）技术受到了计算机视觉和文档分析社区的广泛关注。这些社区已经开发出几种解决方案，无论是基于GAN还是扩散模型的方法，都取得了相当不错的成果。然而，这些策略在面对新型样式时无法实现泛化，并且存在诸如最大输出长度和技术限制等具体问题。 为了解决这些问题，在这项工作中我们提出了一种新的文本图像生成框架，命名为Emuru。我们的方法结合了一个强大的文本图像表示模型（一种变分自编码器）与一个自回归的Transformer。这种方法能够基于文字内容和风格示例来生成带有特定字体或手写样式的文本图像。我们仅使用包含超过10万种打字机和书法字体的合成多样化英语文本数据集训练我们的模型，这使得它能够在零样本情况下再现未见过的样式（包括字体和用户的手写）。据我们所知，Emuru是第一个用于HTG的自回归模型，并且也是首个专门设计以泛化到新风格的模型。此外，我们的模型生成的图像不包含背景伪影，更容易应用于下游任务。 在打字机文本及手写文本（任意长度）的图像生成场景中进行的广泛评估显示了我们方法的有效性。
URL
https://arxiv.org/abs/2503.17074
