Timezone: »

Communication-Efficient Actor-Critic Methods for Homogeneous Markov Games
Dingyang Chen · Yile Li · Qi Zhang
Event URL: https://openreview.net/forum?id=GHN7y7G_Vyb »

Recent success in cooperative multi-agent reinforcement learning (MARL) relies on centralized training and policy sharing. Centralized training eliminates the issue of non-stationarity MARL yet induces large communication costs, and policy sharing is empirically crucial to efficient learning in certain tasks yet lacks theoretical justification. In this paper, we formally characterize a subclass of cooperative Markov games where agents exhibit a certain level of homogeneity such that policy sharing provably incurs no suboptimality. This enables us to develop the first consensus-based decentralized actor-critic method where the consensus update is applied to both the actors and the critics while ensuring convergence. We also develop practical algorithms based on our decentralized actor-critic method to reduce the communication cost during training, while still yielding policies comparable with centralized training.

Author Information

Dingyang Chen (University of South Carolina)
Yile Li (Shanghai Jiao Tong University)
Qi Zhang (University of Michigan)

More from the Same Authors