Abstract
Diffusion models have become widely adopted in image completion tasks, with text prompts commonly employed to ensure semantic coherence by providing high-level guidance. However, a persistent challenge arises when an object is partially obscured in the damaged region, yet its remaining parts are still visible in the background. While text prompts offer semantic direction, they often fail to precisely recover fine-grained structural details, such as the object's overall posture, ensuring alignment with the visible object information in the background. This limitation stems from the inability of text prompts to provide pixel-level specificity. To address this, we propose supplementing text-based guidance with a novel visual aid: a casual sketch, which can be roughly drawn by anyone based on visible object parts. This sketch supplies critical structural cues, enabling the generative model to produce an object structure that seamlessly integrates with the existing background. We introduce the Visual Sketch Self-Aware (VSSA) model, which integrates the casual sketch into each iterative step of the diffusion process, offering distinct advantages for partially corrupted scenarios. By blending sketch-derived features with those of the corrupted image, and leveraging text prompt guidance, the VSSA assists the diffusion model in generating images that preserve both the intended object semantics and structural consistency across the restored objects and original regions. To support this research, we created two datasets, CUB-sketch and MSCOCO-sketch, each combining images, sketches, and text. Extensive qualitative and quantitative experiments demonstrate that our approach outperforms several state-of-the-art methods.
Abstract (translated)
扩散模型在图像补全任务中被广泛采用，通常使用文本提示来确保语义连贯性并提供高层次的指导。然而，当目标物体的部分区域被损坏遮挡时，尽管其剩余部分仍可在背景中看到，这便成为一个持久性的挑战。虽然文本提示提供了语义方向，但它们往往无法精确恢复细微结构细节，例如物体的整体姿态，以确保与背景中可见物体信息的一致性。这一限制源于文本提示无法提供像素级别的具体信息。 为了解决这个问题，我们提出了一种新颖的方法：在基于文本的指导之外添加视觉辅助——一个简单的草图，任何人都可以根据看到的目标物部分轻松绘制出来。这个草图提供了关键的结构线索，使生成模型能够产出与现有背景无缝融合的对象结构。为此，我们引入了“视觉草图自我感知”（Visual Sketch Self-Aware, VSSA）模型，该模型将草图融入扩散过程中的每一步迭代中，在部分损坏的情况下提供独特的优势。 通过结合从草图和受损图像衍生出的特征，并利用文本提示指导，VSSA帮助扩散模型生成既能保持意图对象语义又能确保修复区域与原区结构一致性的图像。为了支持这项研究，我们创建了两个数据集：CUB-sketch 和 MSCOCO-sketch，每个都结合了图像、草图和文本信息。通过广泛的质量和数量的实验验证，我们的方法在多个最先进的技术对比中表现出色。
URL
https://arxiv.org/abs/2503.07047
