Timezone: »

 
Poster
Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
Aaron Sonabend · Junwei Lu · Leo Anthony Celi · Tianxi Cai · Peter Szolovits

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #546
in Poster Session 2 »

Offline Reinforcement Learning (RL) is a promising approach for learning optimal policies in environments where direct exploration is expensive or unfeasible. However, the adoption of such policies in practice is often challenging, as they are hard to interpret within the application context, and lack measures of uncertainty for the learned policy value and its decisions. To overcome these issues, we propose an Expert-Supervised RL (ESRL) framework which uses uncertainty quantification for offline policy learning. In particular, we have three contributions: 1) the method can learn safe and optimal policies through hypothesis testing, 2) ESRL allows for different levels of risk averse implementations tailored to the application context, and finally, 3) we propose a way to interpret ESRL’s policy at every state through posterior distributions, and use this framework to compute off-policy value function posteriors. We provide theoretical guarantees for our estimators and regret bounds consistent with Posterior Sampling for RL (PSRL). Sample efficiency of ESRL is independent of the chosen risk aversion threshold and quality of the behavior policy.

Author Information

Aaron Sonabend (Harvard University)
Junwei Lu
Leo Anthony Celi (Massachusetts Institute of Technology)
Tianxi Cai (Harvard School of Public Health)
Peter Szolovits (MIT)

More from the Same Authors