Timezone: »

 
Poster
Parameter Prediction for Unseen Deep Architectures
Boris Knyazev · Michal Drozdzal · Graham Taylor · Adriana Romero Soriano

Tue Dec 07 04:30 PM -- 06:00 PM (PST) @
in Poster Session 2 »

Deep learning has been successful in automating the design of features in machine learning pipelines. However, the algorithms optimizing neural network parameters remain largely hand-designed and computationally inefficient. We study if we can use deep learning to directly predict these parameters by exploiting the past knowledge of training other networks. We introduce a large-scale dataset of diverse computational graphs of neural architectures - DeepNets-1M - and use it to explore parameter prediction on CIFAR-10 and ImageNet. By leveraging advances in graph neural networks, we propose a hypernetwork that can predict performant parameters in a single forward pass taking a fraction of a second, even on a CPU. The proposed model achieves surprisingly good performance on unseen and diverse networks. For example, it is able to predict all 24 million parameters of a ResNet-50 achieving a 60% accuracy on CIFAR-10. On ImageNet, top-5 accuracy of some of our networks approaches 50%. Our task along with the model and results can potentially lead to a new, more computationally efficient paradigm of training networks. Our model also learns a strong representation of neural architectures enabling their analysis.

Keywords: Deep Learning · Optimization · Machine Learning · Graph Learning · Meta Learning

Author Information

Boris Knyazev (University of Guelph / Vector Institute)
Michal Drozdzal (FAIR)
Graham Taylor (University of Guelph / Vector Institute)
Adriana Romero Soriano (Facebook AI Research)

More from the Same Authors