Timezone: »

 
Accelerating Federated Learning Through Attention on Local Model Updates
Parsa Assadi · Byung Hoon Ahn · Hadi Esmaeilzadeh
Event URL: https://openreview.net/forum?id=jxUrueqBd5B »

Federated learning is used widely for privacy-preserving training. It performs well if the client datasets are both balanced and IID. However, in real-world settings, client datasets are non-IID and imbalanced. They may also experience significant distribution shifts. These non-idealities can hinder the performance of federated learning. To address this challenge, the paper devises an attention-based mechanism that learns to attend to different clients in the context of a reference dataset. The reference dataset is a test dataset in the central server which is used to monitor the performance metric of the model under training. The innovation is that the attention mechanism captures the similarities and patterns of a batch of clients' model drifts (received by the central server in each communication round) in a low dimensional latent space, similar to the way it captures the mutual relation of a batch of words (a sentence). To learn this attention layer, we devise an autoencoder whose input/outputs are the model drifts and its bottleneck is the attention mechanism. The attention weights in the bottleneck are learned by utilizing the attention-based autoencoder as a network to reconstruct the model drift on reference dataset, from the batch of received model drifts from clients in each communication round. The learned attention weights effectively capture clusters and similarities amongst the clients’ datasets. The empirical studies with MNIST, FashionMNIST, and CIFAR10 under a non-IID federated learning setup show that our attention-based autoencoder can identify the cluster of similar clients. Then the central server can use the clustering results to devise a better policy for choosing participants clients in each communication round, thereby reducing the communication rounds by up to 75% on MNIST and FashionMNIST, and 45% on CIFAR10 compared to FedAvg.

Author Information

Parsa Assadi (University of California, San Diego)
Byung Hoon Ahn (UC San Diego)
Hadi Esmaeilzadeh (Protopia AI / University of California San Diego)

More from the Same Authors

  • 2021 : Neuroweaver: Towards a Platform for Designing Translatable Intelligent Closed-loop Neuromodulation Systems »
    Parisa Sarikhani · Hao-Lun Hsu · Sean Kinzer · Hadi Esmaeilzadeh · Babak Mahmoudi
  • 2021 : Protopia AI: Taking on the Missing Link in AI Privacy and Data Protection »
    Byung Hoon Ahn · DoangJoo Synn · Masih Derkani · Eiman Ebrahimi · Hadi Esmaeilzadeh
  • 2019 : Poster Session »
    Clement Canonne · Kwang-Sung Jun · Seth Neel · Di Wang · Giuseppe Vietri · Liwei Song · Jonathan Lebensold · Huanyu Zhang · Lovedeep Gondara · Ang Li · FatemehSadat Mireshghallah · Jinshuo Dong · Anand D Sarwate · Antti Koskela · Joonas Jälkö · Matt Kusner · Dingfan Chen · Mi Jung Park · Ashwin Machanavajjhala · Jayashree Kalpathy-Cramer · · Vitaly Feldman · Andrew Tomkins · Hai Phan · Hossein Esfandiari · Mimansa Jaiswal · Mrinank Sharma · Jeff Druce · Casey Meehan · Zhengli Zhao · Hsiang Hsu · Davis Railsback · Abraham Flaxman · · Julius Adebayo · Aleksandra Korolova · Jiaming Xu · Naoise Holohan · Samyadeep Basu · Matthew Joseph · My Thai · Xiaoqian Yang · Ellen Vitercik · Michael Hutchinson · Chenghong Wang · Gregory Yauney · Yuchao Tao · Chao Jin · Si Kai Lee · Audra McMillan · Rauf Izmailov · Jiayi Guo · Siddharth Swaroop · Tribhuvanesh Orekondy · Hadi Esmaeilzadeh · Kevin Procopio · Alkis Polyzotis · Jafar Mohammadi · Nitin Agrawal
  • 2018 : Poster Session (All Posters) »
    Artemiy Margaritov · Ravichandra Addanki · Hamidreza Mahyar · GUO ZHANG · avani wildani · Hadi Esmaeilzadeh · Dmitrii Ustiugov · Shaileshh Bojja Venkatakrishnan · Fabian Ruffy Varga · adit bhardwaj · Tatiana Shpeisman