Sparse-to-Sparse Training of Diffusion Models

Diffusion models (DMs) are a class of generative models that have achieved state-of-the-art performance in a variety of image synthesis tasks, and have shown promise in other domains. DMs have stable training dynamics, and produce diverse high-quality samples. However, both their training and inference stages demand significant computational costs. Sparse-to-sparse training has emerged as a paradigm to mitigate the computational costs associated with deep learning models, by training networks sparsely from scratch. This work introduces sparse-to-sparse training for DMs, with the aim of improving both training and inference efficiency. We study three methods, Static-DM, RigL-DM, and MagRan-DM, across two diffusion architectures, six datasets and several sparsity levels. Our results show that sparse-to-sparse training offers a promising direction for reducing the memory and computational requirements of DMs while maintaining comparable sample quality. The best sparse configuration outperforms the dense baseline in five of the six model–dataset tested. Our experiments also identify effective sparsity levels and prune/regrowth settings, with higher performance generally achieved using dynamic sparse training at 25–50% sparsity.