Generalization of Diffusion Models Arises from a Regularized Representation Space

Diffusion models are known for generating high-quality, diverse images, yet they can also memorize training samples when overfitting to the empirical data distribution, raising privacy and intellectual property concerns. On the other hand, how they generalize by efficiently learning from a finite set of discrete training samples remains an intriguing problem. We analyze this through the lens of representation learning. Using a ReLU denoising autoencoder (DAE) parameterization, we prove that memorized diffusion models essentially learn the raw data matrix for encoding and decoding, yielding spiky and irregular representations. By contrast, generalized diffusion models capture data statistics within clusters, producing more balanced and regularized representations. We validate these insights by examining representation spaces in real-world diffusion models, where similar distinctions emerge. Building on this observation, we propose a method for detecting memorization that applies to both unconditional and conditional text-to-image diffusion models. Taken together, our results underscore that learning good representations is not only necessary but also central to novel and meaningful image generation.