Differential Privacy Harms Neural Collapse

Differentially Private Stochastic Gradient Descent (DP-SGD) is the de facto gold standard for training neural networks with provable privacy guarantees. However, the trajectory of DP-SGD differs from that of its non-private counterpart, with the most visible effect being its impact on utility. In this work, we show that DP-SGD also affects the network's internal representations by preventing Neural Collapse. We demonstrate that this effect is consistent across several architectures and datasets and persists even at high privacy budgets, suggesting fundamentally different learning dynamics.