Timezone: »

 
Poster
Learning curves of generic features maps for realistic datasets with a teacher-student model
Bruno Loureiro · Cedric Gerbelot · Hugo Cui · Sebastian Goldt · Florent Krzakala · Marc Mezard · Lenka Zdeborová

Thu Dec 09 12:30 AM -- 02:00 AM (PST) @
in Poster Session 5 »

Teacher-student models provide a framework in which the typical-case performance of high-dimensional supervised learning can be described in closed form. The assumptions of Gaussian i.i.d. input data underlying the canonical teacher-student model may, however, be perceived as too restrictive to capture the behaviour of realistic data sets. In this paper, we introduce a Gaussian covariate generalisation of the model where the teacher and student can act on different spaces, generated with fixed, but generic feature maps. While still solvable in a closed form, this generalization is able to capture the learning curves for a broad range of realistic data sets, thus redeeming the potential of the teacher-student framework. Our contribution is then two-fold: first, we prove a rigorous formula for the asymptotic training loss and generalisation error. Second, we present a number of situations where the learning curve of the model captures the one of a realistic data set learned with kernel regression and classification, with out-of-the-box feature maps such as random projections or scattering transforms, or with pre-learned ones - such as the features learned by training multi-layer neural networks. We discuss both the power and the limitations of the framework.

Keywords: Deep Learning · Machine Learning · Kernel Methods

Author Information

Bruno Loureiro (EPFL)
Cedric Gerbelot (Ecole Normale Superieure)
Hugo Cui (Swiss Federal Institute of Technology Lausanne)
Sebastian Goldt (International School of Advanced Studies (SISSA), Trieste, Italy)
Florent Krzakala (EPFL)
Marc Mezard (Ecole normale supérieure)
Lenka Zdeborová (EPFL)

More from the Same Authors