Timezone: »

 
Poster
NAS-Bench-x11 and the Power of Learning Curves
Shen Yan · Colin White · Yash Savani · Frank Hutter

Wed Dec 08 04:30 PM -- 06:00 PM (PST) @ Virtual
in Poster Session 4 »

While early research in neural architecture search (NAS) required extreme computational resources, the recent releases of tabular and surrogate benchmarks have greatly increased the speed and reproducibility of NAS research. However, two of the most popular benchmarks do not provide the full training information for each architecture. As a result, on these benchmarks it is not possible to evaluate many types of multi-fidelity algorithms, such as learning curve extrapolation, that require evaluating architectures at arbitrary epochs. In this work, we present a method using singular value decomposition and noise modeling to create surrogate benchmarks, NAS-Bench-111, NAS-Bench-311, and NAS-Bench-NLP11, that output the full training information for each architecture, rather than just the final validation accuracy. We demonstrate the power of using the full training information by introducing a learning curve extrapolation framework to modify single-fidelity algorithms, showing that it leads to improvements over popular single-fidelity algorithms which claimed to be state-of-the-art upon release.

Keywords: Deep Learning

Author Information

Shen Yan (Michigan State University)
Colin White (Abacus.AI)
Yash Savani (Carnegie Mellon University)

I am a machine learning research scientist, mathematician, and engineer with an unquenchable curiosity and a passion for learning. The pervasive patterns and puzzles of our universe fascinate me, and I strive to exercise my repertoire of math, computation, and collaboration skills to solve those puzzles that have the highest positive societal impact.

Frank Hutter (University of Freiburg & Bosch)

Frank Hutter is a Full Professor for Machine Learning at the Computer Science Department of the University of Freiburg (Germany), where he previously was an assistant professor 2013-2017. Before that, he was at the University of British Columbia (UBC) for eight years, for his PhD and postdoc. Frank's main research interests lie in machine learning, artificial intelligence and automated algorithm design. For his 2009 PhD thesis on algorithm configuration, he received the CAIAC doctoral dissertation award for the best thesis in AI in Canada that year, and with his coauthors, he received several best paper awards and prizes in international competitions on machine learning, SAT solving, and AI planning. Since 2016 he holds an ERC Starting Grant for a project on automating deep learning based on Bayesian optimization, Bayesian neural networks, and deep reinforcement learning.

More from the Same Authors