Timezone: »
Poster
Cross-validation Confidence Intervals for Test Error
Pierre Bayle · Alexandre Bayle · Lucas Janson · Lester Mackey
This work develops central limit theorems for cross-validation and consistent estimators of the asymptotic variance under weak stability conditions on the learning algorithm. Together, these results provide practical, asymptotically-exact confidence intervals for k-fold test error and valid, powerful hypothesis tests of whether one learning algorithm has smaller k-fold test error than another. These results are also the first of their kind for the popular choice of leave-one-out cross-validation. In our experiments with diverse learning algorithms, the resulting intervals and tests outperform the most popular alternative methods from the literature.
Author Information
Pierre Bayle (Princeton University)
Alexandre Bayle (Harvard University)
Lucas Janson (Harvard University)
Lester Mackey (Microsoft Research)
More from the Same Authors
-
2020 Poster: Stochastic Stein Discrepancies »
Jackson Gorham · Anant Raj · Lester Mackey -
2020 Poster: Minimax Estimation of Conditional Moment Models »
Nishanth Dikkala · Greg Lewis · Lester Mackey · Vasilis Syrgkanis -
2020 Poster: Inference for Batched Bandits »
Kelly Zhang · Lucas Janson · Susan Murphy -
2019 Poster: Minimum Stein Discrepancy Estimators »
Alessandro Barp · Francois-Xavier Briol · Andrew Duncan · Mark Girolami · Lester Mackey -
2019 Poster: Accelerating Rescaled Gradient Descent: Fast Optimization of Smooth Functions »
Ashia Wilson · Lester Mackey · Andre Wibisono -
2019 Poster: Stochastic Runge-Kutta Accelerates Langevin Monte Carlo and Beyond »
Xuechen Li · Denny Wu · Lester Mackey · Murat Erdogdu -
2019 Spotlight: Stochastic Runge-Kutta Accelerates Langevin Monte Carlo and Beyond »
Xuechen Li · Denny Wu · Lester Mackey · Murat Erdogdu -
2018 Poster: Random Feature Stein Discrepancies »
Jonathan Huggins · Lester Mackey -
2018 Poster: Global Non-convex Optimization with Discretized Diffusions »
Murat Erdogdu · Lester Mackey · Ohad Shamir -
2015 Poster: Measuring Sample Quality with Stein's Method »
Jackson Gorham · Lester Mackey -
2015 Spotlight: Measuring Sample Quality with Stein's Method »
Jackson Gorham · Lester Mackey -
2014 Workshop: High-energy particle physics, machine learning, and the HiggsML data challenge (HEPML) »
Glen Cowan · Balázs Kégl · Kyle Cranmer · Gábor Melis · Tim Salimans · Vladimir Vava Gligorov · Daniel Whiteson · Lester Mackey · Wojciech Kotlowski · Roberto Díaz Morales · Pierre Baldi · Cecile Germain · David Rousseau · Isabelle Guyon · Tianqi Chen