Abstract
Remarkable progress in text-to-image diffusion models has brought a major concern about potentially generating images on inappropriate or trademarked concepts. Concept erasing has been investigated with the goals of deleting target concepts in diffusion models while preserving other concepts with minimal distortion. To achieve these goals, recent concept erasing methods usually fine-tune the cross-attention layers of diffusion models. In this work, we first show that merely updating the cross-attention layers in diffusion models, which is mathematically equivalent to adding \emph{linear} modules to weights, may not be able to preserve diverse remaining concepts. Then, we propose a novel framework, dubbed Concept Pinpoint Eraser (CPE), by adding \emph{nonlinear} Residual Attention Gates (ResAGs) that selectively erase (or cut) target concepts while safeguarding remaining concepts from broad distributions by employing an attention anchoring loss to prevent the forgetting. Moreover, we adversarially train CPE with ResAG and learnable text embeddings in an iterative manner to maximize erasing performance and enhance robustness against adversarial attacks. Extensive experiments on the erasure of celebrities, artistic styles, and explicit contents demonstrated that the proposed CPE outperforms prior arts by keeping diverse remaining concepts while deleting the target concepts with robustness against attack prompts. Code is available at this https URL
Abstract (translated)
文本到图像的扩散模型取得了显著的进步，但也引发了人们对生成不适当或受商标保护的概念图片的担忧。概念擦除技术的研究目标是在删除特定概念的同时尽可能地保留其他概念而不造成重大失真。最近的方法通常通过微调扩散模型中的交叉注意力层来实现这一目的。 然而，在这项工作中，研究者们首先展示了仅仅更新扩散模型中的交叉注意力层（这在数学上等同于向权重中添加线性模块）可能不足以保护多样化存在的其他概念。为了解决这个问题，他们提出了一种新的框架——概念定点擦除器（Concept Pinpoint Eraser, CPE）。该框架通过添加非线性的残差注意门（Residual Attention Gates, ResAGs），选择性地删除目标概念，同时利用注意力锚定损失来防止遗忘，从而保护其他广泛分布的概念。此外，研究者们还采用对抗训练方法迭代式地优化CPE及其可学习的文本嵌入，以最大化擦除性能并增强对对抗攻击的鲁棒性。 通过在名人、艺术风格和显性内容上的大量实验验证了所提出的CPE框架，在删除目标概念的同时能够保持多样化存在的其他概念，并且具有较强的抗干扰能力。代码可在提供的链接处获取。
URL
https://arxiv.org/abs/2506.22806
