Abstract
Translating freehand sketches into photorealistic images remains a fundamental challenge in image synthesis, particularly due to the abstract, sparse, and stylistically diverse nature of sketches. Existing approaches, including GAN-based and diffusion-based models, often struggle to reconstruct fine-grained details, maintain spatial alignment, or adapt across different sketch domains. In this paper, we propose a component-aware, self-refining framework for sketch-to-image generation that addresses these challenges through a novel two-stage architecture. A Self-Attention-based Autoencoder Network (SA2N) first captures localised semantic and structural features from component-wise sketch regions, while a Coordinate-Preserving Gated Fusion (CGF) module integrates these into a coherent spatial layout. Finally, a Spatially Adaptive Refinement Revisor (SARR), built on a modified StyleGAN2 backbone, enhances realism and consistency through iterative refinement guided by spatial context. Extensive experiments across both facial (CelebAMask-HQ, CUFSF) and non-facial (Sketchy, ChairsV2, ShoesV2) datasets demonstrate the robustness and generalizability of our method. The proposed framework consistently outperforms state-of-the-art GAN and diffusion models, achieving significant gains in image fidelity, semantic accuracy, and perceptual quality. On CelebAMask-HQ, our model improves over prior methods by 21% (FID), 58% (IS), 41% (KID), and 20% (SSIM). These results, along with higher efficiency and visual coherence across diverse domains, position our approach as a strong candidate for applications in forensics, digital art restoration, and general sketch-based image synthesis.
Abstract (translated)
将手绘草图转换为逼真的图像仍然是图像合成中的一个基本挑战，尤其是因为草图具有抽象、稀疏和风格多样的特点。现有的方法，包括基于GAN的模型和扩散模型，往往难以重建细粒度细节、保持空间对齐或适应不同的草图领域。在本文中，我们提出了一种组件感知、自我完善的框架，用于解决从草图生成图像时遇到的问题，并通过新颖的两阶段架构来克服这些挑战。首先，基于自注意力机制的自动编码网络（Self-Attention-based Autoencoder Network, SA2N）捕获来自按组件划分的草图区域的局部语义和结构特征；然后，坐标保持门控融合模块（Coordinate-Preserving Gated Fusion, CGF）将这些信息整合成一个连贯的空间布局。最后，基于修改后的StyleGAN2骨干网构建的空间自适应细化修订器（Spatially Adaptive Refinement Revisor, SARR），通过空间上下文引导的迭代改进来增强现实性和一致性。 在面部（CelebAMask-HQ、CUFSF）和非面部（Sketchy、ChairsV2、ShoesV2）数据集上的广泛实验验证了我们方法的鲁棒性与泛化能力。我们的框架持续优于最先进的GAN和扩散模型，在图像保真度、语义准确性及感知质量方面取得了显著提升。在CelebAMask-HQ上，我们的模型相比先前的方法分别提高了21%（FID）、58%（IS）、41%（KID）以及20%（SSIM）。这些成果，加上更高的效率和跨不同领域的视觉连贯性，使我们的方法成为法医学、数字艺术品修复及通用草图生成图像合成应用的有力候选。
URL
https://arxiv.org/abs/2603.09484
