Timezone: »
An overarching goal in machine learning is to build a generalizable model with few samples. To this end, overparameterization has been the subject of immense interest to explain the generalization ability of deep nets even when the size of the dataset is smaller than that of the model. While the prior literature focuses on the classical supervised setting, this paper aims to demystify overparameterization for meta-learning. Here we have a sequence of linear-regression tasks and we ask: (1) Given earlier tasks, what is the optimal linear representation of features for a new downstream task? and (2) How many samples do we need to build this representation? This work shows that surprisingly, overparameterization arises as a natural answer to these fundamental meta-learning questions. Specifically, for (1), we first show that learning the optimal representation coincides with the problem of designing a task-aware regularization to promote inductive bias. We leverage this inductive bias to explain how the downstream task actually benefits from overparameterization, in contrast to prior works on few-shot learning. For (2), we develop a theory to explain how feature covariance can implicitly help reduce the sample complexity well below the degrees of freedom and lead to small estimation error. We then integrate these findings to obtain an overall performance guarantee for our meta-learning algorithm. Numerical experiments on real and synthetic data verify our insights on overparameterized meta-learning.
Author Information
Yue Sun (University of Washington)
I'm a fourth year Ph.D. student in Department of Electrical Engineering, University of Washington, Seattle with Prof. Maryam Fazel. I graduated as Bachelor of Engineering from Department of Electronics Engineering, Tsinghua University, China. I'm interested in statistical machine learning, optimization and signal processing. Currently I'm working on first order algorithms solving optimization problem of non-convex function, and regularized control/reinforcement learning problems. I joined Google in June-September, 2019, on online optimization algorithm applied in video coding.
Adhyyan Narang (University of Washington, Seattle)
Ibrahim Gulluk (Bogazici University)
Samet Oymak (University of California, Riverside)
Maryam Fazel (University of Washington)
More from the Same Authors
-
2022 : On the Global Convergence of the Regularized Generalized Gauss-Newton Algorithm »
Vincent Roulet · Maryam Fazel · Siddhartha Srinivasa · Zaid Harchaoui -
2023 Poster: No-Regret Online Prediction with Strategic Experts »
Omid Sadeghi · Maryam Fazel -
2022 Poster: Learning in Congestion Games with Bandit Feedback »
Qiwen Cui · Zhihan Xiong · Maryam Fazel · Simon Du -
2022 Poster: Near-Optimal Randomized Exploration for Tabular Markov Decision Processes »
Zhihan Xiong · Ruoqi Shen · Qiwen Cui · Maryam Fazel · Simon Du -
2022 Poster: Non-asymptotic and Accurate Learning of Nonlinear Dynamical Systems »
Yahya Sattar · Samet Oymak -
2021 Poster: AutoBalance: Optimized Loss Functions for Imbalanced Data »
Mingchen Li · Xuechen Zhang · Christos Thrampoulidis · Jiasi Chen · Samet Oymak -
2021 Poster: Global Convergence to Local Minmax Equilibrium in Classes of Nonconvex Zero-Sum Games »
Tanner Fiez · Lillian Ratliff · Eric Mazumdar · Evan Faulkner · Adhyyan Narang -
2021 Poster: Label-Imbalanced and Group-Sensitive Classification under Overparameterization »
Ganesh Ramachandra Kini · Orestis Paraskevas · Samet Oymak · Christos Thrampoulidis -
2021 Poster: Selective Sampling for Online Best-arm Identification »
Romain Camilleri · Zhihan Xiong · Maryam Fazel · Lalit Jain · Kevin Jamieson -
2020 Poster: A Single Recipe for Online Submodular Maximization with Adversarial or Stochastic Constraints »
Omid Sadeghi · Prasanna Raut · Maryam Fazel -
2020 Spotlight: A Single Recipe for Online Submodular Maximization with Adversarial or Stochastic Constraints »
Omid Sadeghi · Prasanna Raut · Maryam Fazel -
2019 Poster: Escaping from saddle points on Riemannian manifolds »
Yue Sun · Nicolas Flammarion · Maryam Fazel -
2016 Poster: Designing smoothing functions for improved worst-case competitive ratio in online optimization »
Reza Eghbali · Maryam Fazel -
2016 Poster: Exploiting Tradeoffs for Exact Recovery in Heterogeneous Stochastic Block Models »
Amin Jalali · Qiyang Han · Ioana Dumitriu · Maryam Fazel -
2012 Poster: Structured learning of Gaussian graphical models »
Karthik Mohan · Michael J Chung · Seungyeop Han · Daniela Witten · Su-In Lee · Maryam Fazel