Poster
Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimation
Qiang Liu · Lihong Li · Ziyang Tang · Dengyong Zhou

Wed Dec 5th 05:00 -- 07:00 PM @ Room 517 AB #121
in Wed Poster Session B »

We consider the off-policy estimation problem of estimating the expected reward of a target policy using samples collected by a different behavior policy. Importance sampling (IS) has been a key technique to derive (nearly) unbiased estimators, but is known to suffer from an excessively high variance in long-horizon problems. In the extreme case of in infinite-horizon problems, the variance of an IS-based estimator may even be unbounded. In this paper, we propose a new off-policy estimation method that applies IS directly on the stationary state-visitation distributions to avoid the exploding variance issue faced by existing estimators.Our key contribution is a novel approach to estimating the density ratio of two stationary distributions, with trajectories sampled from only the behavior distribution. We develop a mini-max loss function for the estimation problem, and derive a closed-form solution for the case of RKHS. We support our method with both theoretical and empirical analyses.

Keywords: Density Estimation · Reinforcement Learning and Planning · Reinforcement Learning

Author Information

Qiang Liu (UT Austin)
Lihong Li (Google Brain)
Ziyang Tang (The University of Texas at Austin)
Denny Zhou (Google Brain)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors