Abstract
Deep generative models have advanced text-to-online handwriting generation (TOHG), which aims to synthesize realistic pen trajectories conditioned on textual input and style references. However, most existing methods still primarily focus on character- or word-level generation, resulting in inefficiency and a lack of holistic structural modeling when applied to full text lines. To address these issues, we propose DiffInk, the first latent diffusion Transformer framework for full-line handwriting generation. We first introduce InkVAE, a novel sequential variational autoencoder enhanced with two complementary latent-space regularization losses: (1) an OCR-based loss enforcing glyph-level accuracy, and (2) a style-classification loss preserving writing style. This dual regularization yields a semantically structured latent space where character content and writer styles are effectively disentangled. We then introduce InkDiT, a novel latent diffusion Transformer that integrates target text and reference styles to generate coherent pen trajectories. Experimental results demonstrate that DiffInk outperforms existing state-of-the-art methods in both glyph accuracy and style fidelity, while significantly improving generation efficiency. Code will be made publicly available.
Abstract (translated)
深度生成模型在文本到在线手写生成（TOHG）方面取得了进展，该技术旨在根据文本输入和风格参考合成逼真的笔迹轨迹。然而，大多数现有的方法仍然主要集中在字符级或单词级别的生成上，在应用于整个文本行时会导致效率低下且缺乏整体结构建模。为了应对这些挑战，我们提出了DiffInk，这是首个用于完整文本行手写生成的潜在扩散Transformer框架。 我们首先引入了InkVAE，这是一种新颖的序列变分自编码器，增强了两种互补的潜在空间正则化损失：(1) 基于OCR的损失，强制要求字符级别的准确性；(2) 保持书写风格的风格分类损失。这种双重正则化产生了语义结构化的潜在空间，在其中可以有效地分离出字符内容和写作风格。 然后我们引入了InkDiT，这是一种新颖的潜在扩散Transformer模型，它可以整合目标文本和参考风格来生成连贯的笔迹轨迹。 实验结果表明，DiffInk在字符准确性、风格保真度方面均优于现有的最先进的方法，并且显著提高了生成效率。代码将在公开发布。
URL
https://arxiv.org/abs/2509.23624
