Timezone: »

 
Poster
Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning
Nathan Kallus · Masatoshi Uehara

Wed Dec 11 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #209
in Reinforcement Learning and Planning -- Reinforcement Learning »

Off-policy evaluation (OPE) in both contextual bandits and reinforcement learning allows one to evaluate novel decision policies without needing to conduct exploration, which is often costly or otherwise infeasible. The problem's importance has attracted many proposed solutions, including importance sampling (IS), self-normalized IS (SNIS), and doubly robust (DR) estimates. DR and its variants ensure semiparametric local efficiency if Q-functions are well-specified, but if they are not they can be worse than both IS and SNIS. It also does not enjoy SNIS's inherent stability and boundedness. We propose new estimators for OPE based on empirical likelihood that are always more efficient than IS, SNIS, and DR and satisfy the same stability and boundedness properties as SNIS. On the way, we categorize various properties and classify existing estimators by them. Besides the theoretical guarantees, empirical studies suggest the new estimators provide advantages.

Keywords: Reinforcement Learning · Causal Inference · Probabilistic Methods · Reinforcement Learning and Planning
[ Paper]  [ Poster] 

Author Information

Nathan Kallus (Cornell University)
Masatoshi Uehara (Harvard University)

More from the Same Authors