Timezone: »

 
Poster
Large-Scale Retrieval for Reinforcement Learning
Peter Humphreys · Arthur Guez · Olivier Tieleman · Laurent Sifre · Theophane Weber · Timothy Lillicrap

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #927
in Poster Session 6 »

Effective decision making involves flexibly relating past experiences and relevant contextual information to a novel situation. In deep reinforcement learning (RL), the dominant paradigm is for an agent to amortise information that helps decision-making into its network weights via gradient descent on training losses. Here, we pursue an alternative approach in which agents can utilise large-scale context-sensitive database lookups to support their parametric computations. This allows agents to directly learn in an end-to-end manner to utilise relevant information to inform their outputs. In addition, new information can be attended to by the agent, without retraining, by simply augmenting the retrieval dataset. We study this approach for offline RL in 9x9 Go, a challenging game for which the vast combinatorial state space privileges generalisation over direct matching to past experiences. We leverage fast, approximate nearest neighbor techniques in order to retrieve relevant data from a set of tens of millions of expert demonstration states. Attending to this information provides a significant boost to prediction accuracy and game-play performance over simply using these demonstrations as training trajectories, providing a compelling demonstration of the value of large-scale retrieval in offline RL agents.

Keywords: Reinforcement Learning · Neural Networks · offline RL · retrieval

Author Information

Peter Humphreys (DeepMind)
Arthur Guez (DeepMind)
Olivier Tieleman (DeepMind)
Laurent Sifre (Google DeepMind)
Theophane Weber (DeepMind)
Timothy Lillicrap (DeepMind & UCL)

More from the Same Authors