Timezone: »

 
Federated Progressive Sparsification (Purge-Merge-Tune)+
Dimitris Stripelis · Umang Gupta · Greg Ver Steeg · Jose-Luis Ambite

@
in Federated Learning: Recent Advances and New Challenges »
Event URL: https://openreview.net/forum?id=GLQqPTRrQMx »

We present FedSparsify, a sparsification strategy for federated training based on progressive weight magnitude pruning, which provides several benefits. First, since the size of the network becomes increasingly smaller, computation and communication costs during training are reduced. Second, the models are incrementally constrained to a smaller set of parameters, which facilitates alignment/merging of the local models, and results in improved learning performance at high sparsity. Third, the final sparsified model is significantly smaller, which improves inference efficiency. We analyze FedSparsify's convergence and empirically demonstrate that FedSparsify can learn a subnetwork smaller than a tenth of the size of the original model with the same or better accuracy compared to existing pruning and no-pruning baselines across several challenging federated learning environments. Our approach leads to an average 4-fold inference efficiency speedup and a 15-fold model size reduction over different domains and neural network architectures.

Author Information

Dimitris Stripelis (Information Sciences Institute, USC)
Dimitris Stripelis

Dimitris Stripelis is currently a Ph.D. candidate in Computer Science at the University of Southern California (USC) graduating this Spring (2023) and his thesis is on Federated Machine Learning Systems for Heterogeneous Environments. Dimitris holds a BSc in Computer Science from the Athens University of Economics and Business and an MSc in Computer Science (Data Science specialization) from the University of Southern California.

Umang Gupta (University of Southern California)
Greg Ver Steeg (USC Information Sciences Institute)
Jose-Luis Ambite (University of Southern California)

More from the Same Authors