Geometry-Aware Differentially Private SGD

Gradient clipping is central to DP-SGD, yet its effectiveness depends on how well the clipping geometry matches the gradient distribution. We introduce GeoClip, which addresses this mismatch by finding a coordinate system where standard DP-SGD clipping better aligns with the gradients. We derive this transformation from a convergence bound and estimate it using already-private gradients, requiring no additional privacy budget. Across our experiments, GeoClip consistently improves over baselines, converging faster and achieving comparable accuracy at a lower privacy cost.