Abstract
To prevent unauthorized use of text in images, Scene Text Removal (STR) has become a crucial task. It focuses on automatically removing text and replacing it with a natural, text-less background while preserving significant details such as texture, color, and contrast. Despite its importance in privacy protection, STR faces several challenges, including boundary artifacts, inconsistent texture and color, and preserving correct shadows. Most STR approaches estimate a text region mask to train a model, solving for image translation or inpainting to generate a text-free image. Thus, the quality of the generated image depends on the accuracy of the inpainting mask and the generator's capability. In this work, we leverage the superior capabilities of diffusion models in generating high-quality, consistent images to address the STR problem. We introduce a ControlNet diffusion model, treating STR as an inpainting task. To enhance the model's robustness, we develop a mask pretraining pipeline to condition our diffusion model. This involves training a masked autoencoder (MAE) using a combination of box masks and coarse stroke masks, and fine-tuning it using masks derived from our novel segmentation-based mask refinement framework. This framework iteratively refines an initial mask and segments it using the SLIC and Hierarchical Feature Selection (HFS) algorithms to produce an accurate final text mask. This improves mask prediction and utilizes rich textural information in natural scene images to provide accurate inpainting masks. Experiments on the SCUT-EnsText and SCUT-Syn datasets demonstrate that our method significantly outperforms existing state-of-the-art techniques.
Abstract (translated)
为了防止未经授权使用图像中的文本，场景文本移除（STR）已成为一项关键任务。它专注于自动删除文本并用自然无文字的背景替换，同时保留诸如纹理、颜色和对比度等重要细节。尽管在隐私保护方面具有重要意义，但STR面临多种挑战，包括边界伪影、不一致的纹理和色彩以及保持正确的阴影。大多数STR方法都会估计一个文本区域掩码来训练模型，通过解决图像翻译或补全问题生成无文字的图像。因此，生成图像的质量取决于补全过程掩码的准确性和生成器的能力。在这项工作中，我们利用扩散模型在生成高质量、一致图像方面的优越能力来解决STR问题。我们引入了一个ControlNet扩散模型，将STR视为一种补全任务。为了提高模型的鲁棒性，我们开发了一种掩码预训练管道来对我们的扩散模型进行条件设置。这包括使用框掩码和粗略笔触掩码组合训练一个掩码自动编码器（MAE），并用来自我们新颖分割基础的掩码优化框架产生的掩码对其进行微调。该框架通过迭代地精化初始掩码，并利用SLIC和层次特征选择（HFS）算法进行分段，生成准确的最终文本掩码。这提高了掩码预测精度，并利用自然场景图像中的丰富纹理信息提供精确的补全掩码。在SCUT-EnsText和SCUT-Syn数据集上的实验表明，我们的方法显著优于现有的最先进的技术。
URL
https://arxiv.org/abs/2410.21721
