Abstract
Achieving precise word-level typography control within generated images remains a persistent challenge. To address it, we newly construct a word-level controlled scene text dataset and introduce the Text-Image Alignment (TIA) framework. This framework leverages cross-modal correspondence between text and local image regions provided by grounding models to enhance the Text-to-Image (T2I) model training. Furthermore, we propose WordCon, a hybrid parameter-efficient fine-tuning (PEFT) method. WordCon reparameterizes selective key parameters, improving both efficiency and portability. This allows seamless integration into diverse pipelines, including artistic text rendering, text editing, and image-conditioned text rendering. To further enhance controllability, the masked loss at the latent level is applied to guide the model to concentrate on learning the text region in the image, and the joint-attention loss provides feature-level supervision to promote disentanglement between different words. Both qualitative and quantitative results demonstrate the superiority of our method to the state of the art. The datasets and source code will be available for academic use.
Abstract (translated)
在生成图像中实现精确的单词级排版控制仍然是一个持续性的挑战。为了解决这个问题，我们构建了一个新的以单词级别进行控制的场景文本数据集，并引入了文本-图像对齐（TIA）框架。该框架利用了由定位模型提供的文本与局部图像区域之间的跨模态对应关系，以此来增强从文本到图像（Text-to-Image, T2I）模型的训练效果。此外，我们还提出了WordCon，这是一种混合参数高效的微调（PEFT）方法。WordCon通过对选择的关键参数进行重新参数化，从而在提高效率和便携性的同时改进了这些关键参数的表现。这使得它可以无缝地整合到各种不同的工作流程中，包括艺术文字渲染、文本编辑以及基于图像的文字渲染等方面。 为了进一步增强控制能力，我们在潜在层面上应用了掩码损失（masked loss），以引导模型专注于学习图像中的文字区域，并且联合注意力损失（joint-attention loss）提供了特征级监督来促进不同单词之间的分离。无论是定性还是定量的结果都证明了我们的方法在现有技术中处于领先地位。 我们将提供数据集和源代码供学术使用。
URL
https://arxiv.org/abs/2506.21276
