Differentially Private Clipping-Free Stochastic Sign Descent

Federated learning allows clients to train a model collaboratively while keeping the data decentralized. For real-world applications, two important conditions must be met: first, training must be differentially private so as not to compromise clients' data, and, second, communication costs must be minimized, especially if the model is large. While SignSGD is memory-effective and at the same time reduces communication costs by transmitting only signs of the gradients, it is not differentially private per se. We propose a differentially private modification of SignSGD that fixes this issue. Instead of clipping the gradients and then noising them coordinate-wise to ensure differential privacy, we randomize sign as a mapping from $\mathbb{R}^d$ to $\lbrace\pm1\rbrace^d$. Our tight privacy estimation shows that this approach delivers better privacy guarantees than adding Gaussian or logistic noise to the clipped gradients, both in terms of $(\varepsilon, 0)$ privacy and the upper bound on $\delta$ for a fixed $\varepsilon$ budget. We empirically demonstrate that these theoretical advantages of our scheme lead to a feasible differentially private algorithm for federated learning problems. A privacy audit supports the guarantees of DP-SignSGD and ours, but rejects that of DP-SignLoSGD, matching an error we find in the accounting behind it.