Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search

YAO SHU · Zhongxiang Dai · Zhaoxuan Wu · Bryan Kian Hsiang Low

Hall J #1041

Keywords: [ Hybrid ] [ Training-free ] [ Bayesian optimization ] [ Neural Architecture Search ] [ Neural Tangent Kernel ]

[ Abstract ]
[ Paper [ Poster [ OpenReview
Tue 29 Nov 9 a.m. PST — 11 a.m. PST


Neural architecture search (NAS) has gained immense popularity owing to its ability to automate neural architecture design. A number of training-free metrics are recently proposed to realize NAS without training, hence making NAS more scalable. Despite their competitive empirical performances, a unified theoretical understanding of these training-free metrics is lacking. As a consequence, (a) the relationships among these metrics are unclear, (b) there is no theoretical interpretation for their empirical performances, and (c) there may exist untapped potential in existing training-free NAS, which probably can be unveiled through a unified theoretical understanding. To this end, this paper presents a unified theoretical analysis of gradient-based training-free NAS, which allows us to (a) theoretically study their relationships, (b) theoretically guarantee their generalization performances, and (c) exploit our unified theoretical understanding to develop a novel framework named hybrid NAS (HNAS) which consistently boosts training-free NAS in a principled way. Remarkably, HNAS can enjoy the advantages of both training-free (i.e., the superior search efficiency) and training-based (i.e., the remarkable search effectiveness) NAS, which we have demonstrated through extensive experiments.

Chat is not available.