Timezone: »

 
Poster
Model Selection for Production System via Automated Online Experiments
Zhenwen Dai · Praveen Chandar · Ghazal Fazelnia · Benjamin Carterette · Mounia Lalmas

Wed Dec 09 09:00 AM -- 11:00 AM (PST) @ Poster Session 3 #1046
in Poster Session 4 »

A challenge that machine learning practitioners in the industry face is the task of selecting the best model to deploy in production. As a model is often an intermediate component of a production system, online controlled experiments such as A/B tests yield the most reliable estimation of the effectiveness of the whole system, but can only compare two or a few models due to budget constraints. We propose an automated online experimentation mechanism that can efficiently perform model selection from a large pool of models with a small number of online experiments. We derive the probability distribution of the metric of interest that contains the model uncertainty from our Bayesian surrogate model trained using historical logs. Our method efficiently identifies the best model by sequentially selecting and deploying a list of models from the candidate set that balance exploration-exploitation. Using simulations based on real data, we demonstrate the effectiveness of our method on two different tasks.

Keywords: Deep Learning · Representation Learning · Algorithms -> Relational Learning; Algorithms

Author Information

Zhenwen Dai (Spotify)
Praveen Chandar (Spotify)

Praveen Chandar is a Senior Research Scientist at Spotify working on search and recommendations. His research interests are in machine learning, information retrieval, and recommendation systems with a focus on experimentation and evaluation. Praveen received his Ph.D. from the University of Delaware, working on novelty and diversity aspects of search evaluation. He was previously a Research Staff Member at IBM Research. He has published papers at top conferences including, SIGIR, KDD, WSDM, WWW, CIKM, CHI, and UAI.

Ghazal Fazelnia (Spotify Research)
Benjamin Carterette (Spotify)
Mounia Lalmas (Spotify)

More from the Same Authors