Contrastive Representation Learning for Robust Concept Erasure in Diffusion Models

Concept erasure removes an unwanted or unsafe concept from a pretrained text-to-image diffusion model while preserving its other capabilities. We argue that erasure should be judged by whether the concept survives an adversary, not only by whether it disappears from ordinary prompts. Existing methods remain vulnerable to adversarial prompts and optimized conditioning embeddings, showing that suppression does not ensure robust erasure. We introduce CIRCE (Contrastive Image-Representation Concept Erasure), a contrastive objective on the denoiser's internal representations. It moves target representations away from their values in the frozen model and toward a shared batch anchor while preserving benign representations. CIRCE sets a new best mean on UnlearnCanvas among ten published defenses. For nudity erasure on Stable Diffusion~1.4, our method achieves the lowest worst-case attack success across seven attacks. Its largest advantage is under continuous concept inversion, where it reaches $0.138$ compared with $0.225$ for the next-best method, while ranking second among erased models in retain-quality Elo. Ablations show that this robustness comes primarily from grounding the forget set in images of the concept rather than strengthening a token-only collapse. These results establish representation-level intervention as an effective route to robust concept erasure when the forget data cover the target capability.