Timezone: »
There is significant recent interest to parallelize deep learning algorithms in order to handle the enormous growth in data and model sizes. While most advances focus on model parallelization and engaging multiple computing agents via using a central parameter server, aspect of data parallelization along with decentralized computation has not been explored sufficiently. In this context, this paper presents a new consensus-based distributed SGD (CDSGD) (and its momentum variant, CDMSGD) algorithm for collaborative deep learning over fixed topology networks that enables data parallelization as well as decentralized computation. Such a framework can be extremely useful for learning agents with access to only local/private data in a communication constrained environment. We analyze the convergence properties of the proposed algorithm with strongly convex and nonconvex objective functions with fixed and diminishing step sizes using concepts of Lyapunov function construction. We demonstrate the efficacy of our algorithms in comparison with the baseline centralized SGD and the recently proposed federated averaging algorithm (that also enables data parallelism) based on benchmark datasets such as MNIST, CIFAR-10 and CIFAR-100.
Author Information
Zhanhong Jiang (Iowa State University)
Aditya Balu (Iowa State University)
Chinmay Hegde (Iowa State University)
Soumik Sarkar (Iowa State University)
More from the Same Authors
-
2021 : Cross-Modal Virtual Sensing for Combustion Instability Monitoring »
Tryambak Gangopadhyay · Vikram Ramanan · Chakravarthy S.R. · Soumik Sarkar -
2022 : 3D Reconstruction of Protein Complex Structures Using Synthesized Multi-View AFM Images »
Jaydeep Rade · Soumik Sarkar · Anwesha Sarkar · Adarsh Krishnamurthy -
2022 : Enhancing System-level Safety in Autonomous Driving via Feedback Learning »
Sin Yong Tan · Weisi Fan · Qisai Liu · Tichakorn Wongpiromsarn · Soumik Sarkar -
2022 : DriveCLIP: Zero-shot transfer for distracted driving activity understanding using CLIP »
Md Zahid Hasan · Ameya Joshi · Mohammed Shaiqur Rahman · Venkatachalapathy Archana · Anuj Sharma · Chinmay Hegde · Soumik Sarkar -
2022 : Generative Design of Material Microstructures for Organic Solar Cells using Diffusion Models »
Ethan Herron · Xian Yeow Lee · Aditya Balu · Baskar Ganapathysubramanian · Soumik Sarkar · Adarsh Krishnamurthy -
2022 : Communication-efficient Decentralized Deep Learning »
Fateme Fotouhi · Aditya Balu · Zhanhong Jiang · Yasaman Esfandiari · Salman Jahani · Soumik Sarkar -
2022 : A study of natural robustness of deep reinforcement learning algorithms towards adversarial perturbations »
Qisai Liu · Xian Yeow Lee · Soumik Sarkar -
2021 Poster: Differentiable Spline Approximations »
Minsu Cho · Aditya Balu · Ameya Joshi · Anjana Deva Prasad · Biswajit Khara · Soumik Sarkar · Baskar Ganapathysubramanian · Adarsh Krishnamurthy · Chinmay Hegde -
2019 : Poster Session »
Jonathan Scarlett · Piotr Indyk · Ali Vakilian · Adrian Weller · Partha P Mitra · Benjamin Aubin · Bruno Loureiro · Florent Krzakala · Lenka Zdeborová · Kristina Monakhova · Joshua Yurtsever · Laura Waller · Hendrik Sommerhoff · Michael Moeller · Rushil Anirudh · Shuang Qiu · Xiaohan Wei · Zhuoran Yang · Jayaraman Thiagarajan · Salman Asif · Michael Gillhofer · Johannes Brandstetter · Sepp Hochreiter · Felix Petersen · Dhruv Patel · Assad Oberai · Akshay Kamath · Sushrut Karmalkar · Eric Price · Ali Ahmed · Zahra Kadkhodaie · Sreyas Mohan · Eero Simoncelli · Carlos Fernandez-Granda · Oscar Leong · Wesam Sakla · Rebecca Willett · Stephan Hoyer · Jascha Sohl-Dickstein · Sam Greydanus · Gauri Jagatap · Chinmay Hegde · Michael Kellman · Jonathan Tamir · Nouamane Laanait · Ousmane Dia · Mirco Ravanelli · Jonathan Binas · Negar Rostamzadeh · Shirin Jalali · Tiantian Fang · Alex Schwing · Sébastien Lachapelle · Philippe Brouillard · Tristan Deleu · Simon Lacoste-Julien · Stella Yu · Arya Mazumdar · Ankit Singh Rawat · Yue Zhao · Jianshu Chen · Xiaoyang Li · Hubert Ramsauer · Gabrio Rizzuti · Nikolaos Mitsakos · Dingzhou Cao · Thomas Strohmer · Yang Li · Pei Peng · Gregory Ongie -
2019 : Poster Session »
Eduard Gorbunov · Alexandre d'Aspremont · Lingxiao Wang · Liwei Wang · Boris Ginsburg · Alessio Quaglino · Camille Castera · Saurabh Adya · Diego Granziol · Rudrajit Das · Raghu Bollapragada · Fabian Pedregosa · Martin Takac · Majid Jahani · Sai Praneeth Karimireddy · Hilal Asi · Balint Daroczy · Leonard Adolphs · Aditya Rawal · Nicolas Brandt · Minhan Li · Giuseppe Ughi · Orlando Romero · Ivan Skorokhodov · Damien Scieur · Kiwook Bae · Konstantin Mishchenko · Rohan Anil · Vatsal Sharan · Aditya Balu · Chao Chen · Zhewei Yao · Tolga Ergen · Paul Grigas · Chris Junchi Li · Jimmy Ba · Stephen J Roberts · Sharan Vaswani · Armin Eftekhari · Chhavi Sharma -
2018 : Contributed Work »
Thaer Moustafa Dieb · Aditya Balu · Amir H. Khasahmadi · Viraj Shah · Boris Knyazev · Payel Das · Garrett Goh · Georgy Derevyanko · Gianni De Fabritiis · Reiko Hagawa · John Ingraham · David Belanger · Jialin Song · Kim Nicoli · Miha Skalic · Michelle Wu · Niklas Gebauer · Peter Bjørn Jørgensen · Ryan-Rhys Griffiths · Shengchao Liu · Sheshera Mysore · Hai Leong Chieu · Philippe Schwaller · Bart Olsthoorn · Bianca-Cristina Cristescu · Wei-Cheng Tseng · Seongok Ryu · Iddo Drori · Kevin Yang · Soumya Sanyal · Zois Boukouvalas · Rishi Bedi · Arindam Paul · Sambuddha Ghosal · Daniil Bash · Clyde Fare · Zekun Ren · Ali Oskooei · Minn Xuan Wong · Paul Sinz · Théophile Gaudin · Wengong Jin · Paul Leu -
2018 Poster: Online Robust Policy Learning in the Presence of Unknown Adversaries »
Aaron Havens · Zhanhong Jiang · Soumik Sarkar -
2017 Poster: Fast, Sample-Efficient Algorithms for Structured Phase Retrieval »
Gauri Jagatap · Chinmay Hegde -
2016 Poster: Fast recovery from a union of subspaces »
Chinmay Hegde · Piotr Indyk · Ludwig Schmidt