Timezone: »

Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation
Yuta Saito · Shunsuke Aihara · Megumi Matsutani · Yusuke Narita

\textit{Off-policy evaluation} (OPE) aims to estimate the performance of hypothetical policies using data generated by a different policy. Because of its huge potential impact in practice, there has been growing research interest in this field. There is, however, no real-world public dataset that enables the evaluation of OPE, making its experimental studies unrealistic and irreproducible. With the goal of enabling realistic and reproducible OPE research, we present \textit{Open Bandit Dataset}, a public logged bandit dataset collected on a large-scale fashion e-commerce platform, ZOZOTOWN. Our dataset is unique in that it contains a set of \textit{multiple} logged bandit datasets collected by running different policies on the same platform. This enables experimental comparisons of different OPE estimators for the first time. We also develop Python software called \textit{Open Bandit Pipeline} to streamline and standardize the implementation of batch bandit algorithms and OPE. Our open data and software will contribute to fair and transparent OPE research and help the community identify fruitful research directions. We provide extensive benchmark experiments of existing OPE estimators using our dataset and software. The results open up essential challenges and new avenues for future OPE research.

Author Information

Yuta Saito (Cornell University)

I am a fourth year undergraduate at Tokyo Institute of Technology. My research lies at the intersection of machine learning and causal inference called counterfactual machine learning. I am interested in the counterfactual nature of logged bandit feedback obtained from interactive systems, and ways of using biased real-world datasets to assist better decision making. Most recently, I have been focusing on the intersection of counterfactual machine learning and unsupervised domain adaptation.

Shunsuke Aihara
Megumi Matsutani
Yusuke Narita (Yale University)

More from the Same Authors

  • 2019 : Coffee break, posters, and 1-on-1 discussions »
    Julius von Kügelgen · David Rohde · Candice Schumann · Grace Charles · Victor Veitch · Vira Semenova · Mert Demirer · Vasilis Syrgkanis · Suraj Nair · Aahlad Puli · Masatoshi Uehara · Aditya Gopalan · Yi Ding · Ignavier Ng · Khashayar Khosravi · Eli Sherman · Shuxi Zeng · Aleksander Wieczorek · Hao Liu · Kyra Gan · Jason Hartford · Miruna Oprescu · Alexander D'Amour · Jörn Boehnke · Yuta Saito · Théophile Griveau-Billion · Chirag Modi · Shyngys Karimov · Jeroen Berrevoets · Logan Graham · Imke Mayer · Dhanya Sridhar · Issa Dahabreh · Alan Mishler · Duncan Wadsworth · Khizar Qureshi · Rahul Ladhania · Gota Morishita · Paul Welle
  • 2019 : Poster Spotlights »
    Hongseok Namkoong · Marie Charpignon · Maja Rudolph · Amanda Coston · Yuta Saito · Paramveer Dhillon · Alexander Markham