Poster
Safe and Efficient Off-Policy Reinforcement Learning
Remi Munos · Tom Stepleton · Anna Harutyunyan · Marc Bellemare

Mon Dec 5th 06:00 -- 09:30 PM @ Area 5+6+7+8 #151 #None
in Monday Posters »

In this work, we take a fresh look at some old and new algorithms for off-policy, return-based reinforcement learning. Expressing these in a common form, we derive a novel algorithm, Retrace(lambda), with three desired properties: (1) it has low variance; (2) it safely uses samples collected from any behaviour policy, whatever its degree of "off-policyness"; and (3) it is efficient as it makes the best use of samples collected from near on-policy behaviour policies. We analyse the contractive nature of the related operator under both off-policy policy evaluation and control settings and derive online sample-based algorithms. We believe this is the first return-based off-policy control algorithm converging a.s. to Q* without the GLIE assumption (Greedy in the Limit with Infinite Exploration). As a corollary, we prove the convergence of Watkins' Q(lambda), which was an open problem since 1989. We illustrate the benefits of Retrace(lambda) on a standard suite of Atari 2600 games.

Keywords: Reinforcement Learning Algorithms

Author Information

Remi Munos (Google DeepMind)
Tom Stepleton (Google DeepMind)
Anna Harutyunyan (Vrije Universiteit Brussel)
Marc Bellemare (Google DeepMind)

More from the Same Authors