Timezone: »

 
Poster
Importance Resampling for Off-policy Prediction
Matthew Schlegel · Wesley Chung · Daniel Graves · Jian Qian · Martha White

Tue Dec 10 05:30 PM -- 07:30 PM (PST) @ East Exhibition Hall B + C #206
in Reinforcement Learning and Planning -- Reinforcement Learning »

Importance sampling (IS) is a common reweighting strategy for off-policy prediction in reinforcement learning. While it is consistent and unbiased, it can result in high variance updates to the weights for the value function. In this work, we explore a resampling strategy as an alternative to reweighting. We propose Importance Resampling (IR) for off-policy prediction, which resamples experience from a replay buffer and applies standard on-policy updates. The approach avoids using importance sampling ratios in the update, instead correcting the distribution before the update. We characterize the bias and consistency of IR, particularly compared to Weighted IS (WIS). We demonstrate in several microworlds that IR has improved sample efficiency and lower variance updates, as compared to IS and several variance-reduced IS strategies, including variants of WIS and V-trace which clips IS ratios. We also provide a demonstration showing IR improves over IS for learning a value function from images in a racing car simulator.

Keywords: Reinforcement Learning and Planning -> Reinforcement Learning ·

Author Information

Matthew Schlegel (University of Alberta)

An AI and coffee enthusiast with research experience in RL and ML. Currently pursuing a PhD at the University of Alberta! Excited about off-policy policy evaluation, general value functions, understanding the behavior of artificial neural networks, and cognitive science (specifically cognitive neuroscience).

Wes Chung (McGill University)
Daniel Graves (Huawei Technologies Canada)
Jian Qian (University of Alberta)
Martha White (University of Alberta)

More from the Same Authors