Timezone: »

 
Poster
Coordinated Proximal Policy Optimization
Zifan Wu · Chao Yu · Deheng Ye · Junge Zhang · haiyin piao · Hankz Hankui Zhuo

Wed Dec 08 12:30 AM -- 02:00 AM (PST) @
in Poster Session 3 »

We present Coordinated Proximal Policy Optimization (CoPPO), an algorithm that extends the original Proximal Policy Optimization (PPO) to the multi-agent setting. The key idea lies in the coordinated adaptation of step size during the policy update process among multiple agents. We prove the monotonicity of policy improvement when optimizing a theoretically-grounded joint objective, and derive a simplified optimization objective based on a set of approximations. We then interpret that such an objective in CoPPO can achieve dynamic credit assignment among agents, thereby alleviating the high variance issue during the concurrent update of agent policies. Finally, we demonstrate that CoPPO outperforms several strong baselines and is competitive with the latest multi-agent PPO method (i.e. MAPPO) under typical multi-agent settings, including cooperative matrix games and the StarCraft II micromanagement tasks.

Keywords: Optimization · Reinforcement Learning and Planning

Author Information

Zifan Wu (Sun Yat-sen University)
Chao Yu (Sun Yat-sen University)
Deheng Ye (Tencent)
Junge Zhang (CASIA)
haiyin piao (Northwestern Polytechnical University)
Hankz Hankui Zhuo (Sun Yat-sen University)

More from the Same Authors