Timezone: »

 
Poster
Modeling the Machine Learning Multiverse
Samuel J. Bell · Onno Kampman · Jesse Dodge · Neil Lawrence

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #119
in Poster Session 1 »

Amid mounting concern about the reliability and credibility of machine learning research, we present a principled framework for making robust and generalizable claims: the multiverse analysis. Our framework builds upon the multiverse analysis introduced in response to psychology's own reproducibility crisis. To efficiently explore high-dimensional and often continuous ML search spaces, we model the multiverse with a Gaussian Process surrogate and apply Bayesian experimental design. Our framework is designed to facilitate drawing robust scientific conclusions about model performance, and thus our approach focuses on exploration rather than conventional optimization. In the first of two case studies, we investigate disputed claims about the relative merit of adaptive optimizers. Second, we synthesize conflicting research on the effect of learning rate on the large batch training generalization gap. For the machine learning community, the multiverse analysis is a simple and effective technique for identifying robust claims, for increasing transparency, and a step toward improved reproducibility.

Keywords: reproducibility · transparency · replication · multiverse analysis · generalization gap · adaptive optimizers · batch size
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Samuel J. Bell (University of Cambridge)
Samuel J. Bell

Sam applies theoretical approaches, practical methods and metascientific ideas from experimental psychology to understanding machine learning systems, with a particular focus on neural network behaviour. He is primarily interested in machine learning fairness, robustness, and research reproducibility. He is a machine learning PhD student in the ML@CL group at the University of Cambridge (Queens’ College), supervised by Prof. Neil Lawrence. Sam was a visiting student at The Alan Turing Institute during the 2021/2 academic year, and has recently worked with the socially-responsible AI team at FAIR (Meta AI) Paris. During his master’s he worked on deep learning and natural language processing at the Cambridge Computer Laboratory. He did his bachelor’s in computer science at the University of Manchester, graduating 2013. In between, Sam has also simulated financial crises in market risk at Goldman Sachs, built new retail banks at Thought Machine, and developed next generation credit scores at Credit Kudos.

Onno Kampman (University of Cambridge)
Jesse Dodge (Allen Institute for AI)
Neil Lawrence (University of Cambridge)

More from the Same Authors