Poster
A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning
Marc Lanctot · Vinicius Zambaldi · Audrunas Gruslys · Angeliki Lazaridou · Karl Tuyls · Julien Perolat · David Silver · Thore Graepel

Wed Dec 6th 06:30 -- 10:30 PM @ Pacific Ballroom #203 #None
in Posters Wed »

There has been a resurgence of interest in multiagent reinforcement learning (MARL), due partly to the recent success of deep neural networks. The simplest form of MARL is independent reinforcement learning (InRL), where each agent treats all of its experience as part of its (non stationary) environment. In this paper, we first observe that policies learned using InRL can overfit to the other agents' policies during training, failing to sufficiently generalize during execution. We introduce a new metric, joint-policy correlation, to quantify this effect. We describe a meta-algorithm for general MARL, based on approximate best responses to mixtures of policies generated using deep reinforcement learning, and empirical game theoretic analysis to compute meta-strategies for policy selection. The meta-algorithm generalizes previous algorithms such as InRL, iterated best response, double oracle, and fictitious play. Then, we propose a scalable implementation which reduces the memory requirement using decoupled meta-solvers. Finally, we demonstrate the generality of the resulting policies in three partially observable settings: gridworld coordination problems, emergent language games, and poker.

Keywords: Reinforcement Learning · Biologically Plausible Deep Networks · Multi-Agent RL · Game Theory and Computational Economics

Author Information

Marc Lanctot (DeepMind)
Vinicius Zambaldi (Deepmind)
Audrunas Gruslys (Google DeepMind)
Angeliki Lazaridou (DeepMind)
Karl Tuyls (DeepMind)
Julien Perolat (DeepMind)
David Silver (DeepMind)
Thore Graepel (DeepMind)

More from the Same Authors