Abstract
Scene text editing, a subfield of image editing, requires modifying texts in images while preserving style consistency and visual coherence with the surrounding environment. While diffusion-based methods have shown promise in text generation, they still struggle to produce high-quality results. These methods often generate distorted or unrecognizable characters, particularly when dealing with complex characters like Chinese. In such systems, characters are composed of intricate stroke patterns and spatial relationships that must be precisely maintained. We present GlyphMastero, a specialized glyph encoder designed to guide the latent diffusion model for generating texts with stroke-level precision. Our key insight is that existing methods, despite using pretrained OCR models for feature extraction, fail to capture the hierarchical nature of text structures - from individual strokes to stroke-level interactions to overall character-level structure. To address this, our glyph encoder explicitly models and captures the cross-level interactions between local-level individual characters and global-level text lines through our novel glyph attention module. Meanwhile, our model implements a feature pyramid network to fuse the multi-scale OCR backbone features at the global-level. Through these cross-level and multi-scale fusions, we obtain more detailed glyph-aware guidance, enabling precise control over the scene text generation process. Our method achieves an 18.02\% improvement in sentence accuracy over the state-of-the-art multi-lingual scene text editing baseline, while simultaneously reducing the text-region Fréchet inception distance by 53.28\%.
Abstract (translated)
场景文本编辑是图像编辑的一个子领域，它要求在保留风格一致性和周围环境视觉连贯性的前提下修改图片中的文字。虽然基于扩散的方法在文本生成方面显示出潜力，但它们仍然难以产生高质量的结果。这些方法经常生成扭曲或无法辨认的字符，尤其是在处理像中文这样复杂的字符时。在这种系统中，字符由精细的笔画模式和空间关系构成，必须精确维护。 我们提出了GlyphMastero，这是一种专门设计用于引导潜在扩散模型以实现笔画级别精度的文字生成技术的字体编码器。我们的关键见解是现有的方法虽然使用了预训练的OCR（光学字符识别）模型进行特征提取，但未能捕捉文本结构的层次性——从单个笔画到笔画级别的交互再到整个字符级别的结构。为了解决这个问题，我们的字体编码器通过我们新颖的字体注意模块明确地建模并捕获局部级别个体字符与全局级别文字行之间的跨层级互动。 同时，我们的模型实现了一个特征金字塔网络来融合多尺度OCR骨干特性在全局层面的信息。通过这些跨层级和多层次的融合，我们获得了更为详细的文字感知指导，从而能够对场景文本生成过程进行更精确的控制。 我们的方法相对于最先进的多语言场景文本编辑基线，在句子准确度方面提高了18.02%，同时将文本区域Fréchet inception距离减少了53.28%。
URL
https://arxiv.org/abs/2505.04915
