Differential Privacy harms Neural Collapse

Differentially Private Stochastic Gradient Descent (DP-SGD) is the de facto gold standard for training neural networks with provable privacy guarantees. However, the trajectory of DP-SGD differs from that of its non-private counterpart, with the most visible effect being its impact on utility. In this work, we show that DP-SGD also affects the network's internal representations by preventing Neural Collapse, a phenomenon well-established in non-private training, where the last layer of the network exhibits a particular geometry that maximizes the separation between classes. We demonstrate that Neural Collapse consistently disappears across several architectures and datasets, and that this effect persists even at high privacy budgets, suggesting fundamentally different learning dynamics.