Abstract
Scene text editing aims to modify text content within scene images while maintaining style consistency. Traditional methods achieve this by explicitly disentangling style and content from the source image and then fusing the style with the target content, while ensuring content consistency using a pre-trained recognition model. Despite notable progress, these methods suffer from complex pipelines, leading to suboptimal performance in complex scenarios. In this work, we introduce Recognition-Synergistic Scene Text Editing (RS-STE), a novel approach that fully exploits the intrinsic synergy of text recognition for editing. Our model seamlessly integrates text recognition with text editing within a unified framework, and leverages the recognition model's ability to implicitly disentangle style and content while ensuring content consistency. Specifically, our approach employs a multi-modal parallel decoder based on transformer architecture, which predicts both text content and stylized images in parallel. Additionally, our cyclic self-supervised fine-tuning strategy enables effective training on unpaired real-world data without ground truth, enhancing style and content consistency through a twice-cyclic generation process. Built on a relatively simple architecture, \mymodel achieves state-of-the-art performance on both synthetic and real-world benchmarks, and further demonstrates the effectiveness of leveraging the generated hard cases to boost the performance of downstream recognition tasks. Code is available at this https URL.
Abstract (translated)
场景文本编辑的目标是在保持风格一致性的前提下修改场景图像中的文字内容。传统方法通过从源图中显式地分离出样式和内容，然后将这些样式与目标内容融合来实现这一目的，并使用预训练的识别模型确保内容的一致性。尽管取得了显著的进步，但这些方法由于复杂的管道而难以在复杂的情况下表现出色。 在这项工作中，我们引入了一种新的方法——协同识别场景文本编辑（Recognition-Synergistic Scene Text Editing, RS-STE），它充分利用了文字识别与编辑之间的内在联系。我们的模型在一个统一的框架内无缝地结合了文本识别和文本编辑，并利用了识别模型在隐式分离样式和内容的同时保持内容一致性的能力。具体而言，我们采用了一种基于变压器架构的多模态并行解码器，可以同时预测文本内容和风格化图像。 此外，我们的循环自我监督微调策略能够在没有真实标签的情况下有效训练未配对的真实世界数据，并通过两次循环生成过程增强样式和内容的一致性。在相对简单的架构基础上，\mymodel 在合成和真实世界的基准测试中均达到了最先进的性能，并进一步证明了利用生成的难例来提升下游识别任务性能的有效性。 代码可以在提供的链接地址获取。
URL
https://arxiv.org/abs/2503.08387
