Detecting and Preventing Memorization in Diffusion Models via Fisher-Rao Guidance

It is well known that overtraining diffusion models on finite datasets can drive a transition from learning a smooth distribution to reproducing individual training examples. We study this transition through the responsibilities of the training samples under the finite-sample score. Their Fisher-Rao geometry yields a local notion of atomic commitment through an energy given by the normalized posterior covariance of competing training atoms, which is large when several atoms remain plausible and vanishes when a single atom dominates. This characterization yields both a local diagnostic of memorization and an efficiently implementable inference-time intervention: by guiding trajectories toward regions of higher Fisher-Rao energy, we discourage premature commitment to individual training examples. Experiments on both a controlled sinusoidal manifold and a deliberately memorized CelebA-HQ and CIFAR-10 diffusion models show that the proposed guidance substantially reduces attraction toward memorized training atoms while largely preserving the underlying data-manifold structure. An anonymized version of the code is available in the GitHub repository https://anonymous.4open.science/r/denoising_memorization-E4B7.