Timezone: »

 
Poster
NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks
Renbo Tu · Nicholas Roberts · Misha Khodak · Junhong Shen · Frederic Sala · Ameet Talwalkar

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #1029
in Poster Session 1 »

Most existing neural architecture search (NAS) benchmarks and algorithms prioritize well-studied tasks, e.g. image classification on CIFAR or ImageNet. This makes the performance of NAS approaches in more diverse areas poorly understood. In this paper, we present NAS-Bench-360, a benchmark suite to evaluate methods on domains beyond those traditionally studied in architecture search, and use it to address the following question: do state-of-the-art NAS methods perform well on diverse tasks? To construct the benchmark, we curate ten tasks spanning a diverse array of application domains, dataset sizes, problem dimensionalities, and learning objectives. Each task is carefully chosen to interoperate with modern CNN-based search methods while possibly being far-afield from its original development domain. To speed up and reduce the cost of NAS research, for two of the tasks we release the precomputed performance of 15,625 architectures comprising a standard CNN search space. Experimentally, we show the need for more robust NAS evaluation of the kind NAS-Bench-360 enables by showing that several modern NAS procedures perform inconsistently across the ten tasks, with many catastrophically poor results. We also demonstrate how NAS-Bench-360 and its associated precomputed results will enable future scientific discoveries by testing whether several recent hypotheses promoted in the NAS literature hold on diverse tasks. NAS-Bench-360 is hosted at https://nb360.ml.cmu.edu.

Keywords: Neural Architecture Search · Automated Machine Learning · diverse tasks

Author Information

Renbo Tu (University of Toronto)
Nicholas Roberts (University of Wisconsin-Madison)

I am a Ph.D. student in CS at University of Wisconsin – Madison where I am advised by Fred Sala. Before that, I had the pleasure of working with Ameet Talwalkar and Zack Lipton during my MS at Carnegie Mellon University. As an undergraduate, I was extremely fortunate to work with both Sanjoy Dasgupta and Gary Cottrell at the University of California, San Diego. Before that, I was a community college student at Fresno City College, where I was lucky enough to learn calculus, linear algebra, AND C++ from Greg Jamison.

Misha Khodak (CMU)
Junhong Shen (Carnegie Mellon University)
Frederic Sala (University of Wisconsin, Madison)
Ameet Talwalkar (CMU)

More from the Same Authors