Timezone: »

 
Poster
Towards Sample-efficient Overparameterized Meta-learning
Yue Sun · Adhyyan Narang · Ibrahim Gulluk · Samet Oymak · Maryam Fazel

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @
in Poster Session 1 »

An overarching goal in machine learning is to build a generalizable model with few samples. To this end, overparameterization has been the subject of immense interest to explain the generalization ability of deep nets even when the size of the dataset is smaller than that of the model. While the prior literature focuses on the classical supervised setting, this paper aims to demystify overparameterization for meta-learning. Here we have a sequence of linear-regression tasks and we ask: (1) Given earlier tasks, what is the optimal linear representation of features for a new downstream task? and (2) How many samples do we need to build this representation? This work shows that surprisingly, overparameterization arises as a natural answer to these fundamental meta-learning questions. Specifically, for (1), we first show that learning the optimal representation coincides with the problem of designing a task-aware regularization to promote inductive bias. We leverage this inductive bias to explain how the downstream task actually benefits from overparameterization, in contrast to prior works on few-shot learning. For (2), we develop a theory to explain how feature covariance can implicitly help reduce the sample complexity well below the degrees of freedom and lead to small estimation error. We then integrate these findings to obtain an overall performance guarantee for our meta-learning algorithm. Numerical experiments on real and synthetic data verify our insights on overparameterized meta-learning.

Keywords: Theory · Machine Learning · Meta Learning · Representation Learning · Few Shot Learning

Author Information

Yue Sun (University of Washington)

I'm a fourth year Ph.D. student in Department of Electrical Engineering, University of Washington, Seattle with Prof. Maryam Fazel. I graduated as Bachelor of Engineering from Department of Electronics Engineering, Tsinghua University, China. I'm interested in statistical machine learning, optimization and signal processing. Currently I'm working on first order algorithms solving optimization problem of non-convex function, and regularized control/reinforcement learning problems. I joined Google in June-September, 2019, on online optimization algorithm applied in video coding.

Adhyyan Narang (University of Washington, Seattle)
Ibrahim Gulluk (Bogazici University)
Samet Oymak (University of California, Riverside)
Maryam Fazel (University of Washington)

More from the Same Authors