Timezone: »

 
Oral
PyGlove: Symbolic Programming for Automated Machine Learning
Daiyi Peng · Xuanyi Dong · Esteban Real · Mingxing Tan · Yifeng Lu · Gabriel Bender · Hanxiao Liu · Adam Kraft · Chen Liang · Quoc V Le

Wed Dec 09 06:30 PM -- 06:45 PM (PST) @ Orals & Spotlights: Graph/Meta Learning/Software
in Orals & Spotlights Track 23: Graph/Meta Learning/Software »

Neural networks are sensitive to hyper-parameter and architecture choices. Automated Machine Learning (AutoML) is a promising paradigm for automating these choices. Current ML software libraries, however, are quite limited in handling the dynamic interactions among the components of AutoML. For example, efficient NAS algorithms, such as ENAS and DARTS, typically require an implementation coupling between the search space and search algorithm, the two key components in AutoML. Furthermore, implementing a complex search flow, such as searching architectures within a loop of searching hardware configurations, is difficult. To summarize, changing the search space, search algorithm, or search flow in current ML libraries usually requires a significant change in the program logic.

In this paper, we introduce a new way of programming AutoML based on symbolic programming. Under this paradigm, ML programs are mutable, thus can be manipulated easily by another program. As a result, AutoML can be reformulated as an automated process of symbolic manipulation. With this formulation, we decouple the triangle of the search algorithm, the search space and the child program. This decoupling makes it easy to change the search space and search algorithm (without and with weight sharing), as well as to add search capabilities to existing code and implement complex search flows. We then introduce PyGlove, a new Python library that implements this paradigm. Through case studies on ImageNet and NAS-Bench-101, we show that with PyGlove users can easily convert a static program into a search space, quickly iterate on the search spaces and search algorithms, and craft complex search flows to achieve better results.

Author Information

Daiyi Peng (Google)
Xuanyi Dong (University of Technology Sydney)

Xuanyi Dong is 3-rd year Ph.D. student of Centre for Artificial Intelligence at University of Technology Sydney. His research topic is automated deep learning, especially neural architecture search and its application to computer vision. He has published almost 20 papers on top-tiered conferences and journals including CVPR, ICCV, NuerIPS, T-PAMI. He was elected as one of the 2019 Google Ph.D. Fellows.

Esteban Real (Google Brain)
Mingxing Tan (Google Brain)
Yifeng Lu
Gabriel Bender (Google Brain)
Hanxiao Liu (Google Brain)
Adam Kraft (Google)
Chen Liang (Google Brain)
Quoc V Le (Google)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors