Timezone: »

 
Optimal and Adaptive Off-policy Evaluation in Contextual Bandits
Yu-Xiang Wang

Sat Dec 10 02:00 AM -- 02:30 AM (PST) @
in "What If?" Inference and Learning of Hypothetical and Counterfactual Interventions in Complex Systems »

We consider the problem of off-policy evaluation—estimating the value of a target policy using data collected by another policy—under the contextual bandit model. We establish a minimax lower bound on the mean squared error (MSE), and show that it is matched up to constant factors by the inverse propensity scoring (IPS) estimator. Since in the multi-armed bandit problem the IPS is suboptimal, our result highlights the difficulty of the contextual setting with non-degenerate context distributions. We further consider improvements on this minimax MSE bound, given access to a reward model. We show that the existing doubly robust approach, which utilizes such a reward model, may continue to suffer from high variance even when the reward model is perfect. We propose a new estimator called SWITCH which more effectively uses the reward model and achieves a superior bias-variance tradeoff compared with prior work. We prove an upper bound on its MSE and demonstrate its benefits empirically on a diverse collection of datasets, often seeing orders of magnitude improvements over a number of baselines.

Author Information

Yu-Xiang Wang (UC Santa Barbara)

More from the Same Authors