Timezone: »
Multi-simulator training has contributed to the recent success of Deep Reinforcement Learning (Deep RL) by stabilizing learning and allowing for higher training throughputs. In this work, we propose Gossip-based Actor-Learner Architectures (GALA) where several actor-learners (such as A2C agents) are organized in a peer-to-peer communication topology, and exchange information through asynchronous gossip in order to take advantage of a large number of distributed simulators. We prove that GALA agents remain within an epsilon-ball of one-another during training when using loosely coupled asynchronous communication. By reducing the amount of synchronization between agents, GALA is more computationally efficient and scalable compared to A2C, its fully-synchronous counterpart. GALA also outperforms A2C, being more robust and sample efficient. We show that we can run several loosely coupled GALA agents in parallel on a single GPU and achieve significantly higher hardware utilization and frame-rates than vanilla A2C at comparable power draws.
Author Information
Mahmoud Assran (McGill University / Facebook AI Research)
Joshua Romoff (McGill University)
Nicolas Ballas (Facebook FAIR)
Joelle Pineau (Facebook)
Mike Rabbat (Facebook FAIR)
More from the Same Authors
-
2021 : Stochastic Polyak Stepsize with a Moving Target »
Robert Gower · Aaron Defazio · Mike Rabbat -
2022 : Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learning »
John Nguyen · Jianyu Wang · Kshitiz Malik · Maziar Sanjabi · Mike Rabbat -
2022 : The Interpolated MVU Mechanism For Communication-efficient Private Federated Learning »
Chuan Guo · Kamalika Chaudhuri · Pierre STOCK · Mike Rabbat -
2018 Poster: Fast Approximate Natural Gradient Descent in a Kronecker Factored Eigenbasis »
Thomas George · César Laurent · Xavier Bouthillier · Nicolas Ballas · Pascal Vincent -
2017 Poster: Hybrid Reward Architecture for Reinforcement Learning »
Harm Van Seijen · Mehdi Fatemi · Romain Laroche · Joshua Romoff · Tavian Barnes · Jeffrey Tsang