Skip to yearly menu bar Skip to main content


Workshop

Mathematics of Modern Machine Learning (M3L)

Zhiyuan Li ⋅ Tengyu Ma ⋅ Surbhi Goel ⋅ Kaifeng Lyu ⋅ Christina Baek ⋅ Bingbin Liu ⋅ Alex Damian ⋅ Aditi Raghunathan
Project Page

Abstract

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles
6:50 AM
Opening Remarks
8:30 AM
Feature Learning in Infinite-Depth Neural Networks
Greg Yang ⋅ Dingli Yu ⋅ Chen Zhu ⋅ Soufiane Hayou
9:00 AM
Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data
Zhiwei Xu ⋅ Yutong Wang ⋅ Spencer Frei ⋅ Gal Vardi ⋅ Wei Hu
9:10 AM
Poster Session
10:10 AM
Lunch Break
12:00 PM
Break
1:00 PM
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
Blake Bordelon ⋅ Lorenzo Noci ⋅ Mufan Li ⋅ Boris Hanin ⋅ Cengiz Pehlevan
1:10 PM
Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP
Zixiang Chen ⋅ Yihe Deng ⋅ Yuanzhi Li ⋅ Quanquan Gu
1:20 PM
In-Context Convergence of Transformers
Yu Huang ⋅ Yuan Cheng ⋅ Yingbin Liang
1:30 PM
Large Catapults in Momentum Gradient Descent with Warmup: An Empirical Study
Prin Phunyaphibarn ⋅ Junghyun Lee ⋅ Bohan Wang ⋅ Huishuai Zhang ⋅ Chulhee Yun
1:40 PM
Linear attention is (maybe) all you need (to understand transformer optimization)
Kwangjun Ahn ⋅ Xiang Cheng ⋅ Minhak Song ⋅ Chulhee Yun ⋅ Ali Jadbabaie ⋅ Suvrit Sra
1:50 PM
Closing Remarks
2:00 PM
Poster Session
A PAC-Bayesian Perspective on the Interpolating Information Criterion
Liam Hodgkinson ⋅ Chris van der Heide ⋅ Robert Salomone ⋅ Fred Roosta ⋅ Michael Mahoney
Graph Neural Networks Benefit from Structural Information Provably: A Feature Learning Perspective
Wei Huang ⋅ Yuan Cao ⋅ Haonan Wang ⋅ Xin Cao ⋅ Taiji Suzuki
Linear attention is (maybe) all you need (to understand transformer optimization)
Kwangjun Ahn ⋅ Xiang Cheng ⋅ Minhak Song ⋅ Chulhee Yun ⋅ Ali Jadbabaie ⋅ Suvrit Sra
Large Catapults in Momentum Gradient Descent with Warmup: An Empirical Study
Prin Phunyaphibarn ⋅ Junghyun Lee ⋅ Bohan Wang ⋅ Huishuai Zhang ⋅ Chulhee Yun
Feature Learning in Infinite-Depth Neural Networks
Greg Yang ⋅ Dingli Yu ⋅ Chen Zhu ⋅ Soufiane Hayou
Variational Classification
Shehzaad Dhuliawala ⋅ Mrinmaya Sachan ⋅ Carl Allen
Spectrum Extraction and Clipping for Implicitly Linear Layers
Ali Ebrahimpour-Boroojeny ⋅ Matus Telgarsky ⋅ Hari Sundaram
Curvature-Dimension Tradeoff for Generalization in Hyperbolic Space
Nicolás Alvarado ⋅ Hans Lobel ⋅ Mircea Petrache
Unveiling the Hessian's Connection to the Decision Boundary
Mahalakshmi Sabanayagam ⋅ Freya Behrens ⋅ Urte Adomaityte ⋅ Anna Dawid
Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks
Zixuan Zhang ⋅ Kaiqi Zhang ⋅ Minshuo Chen ⋅ Yuma Takeda ⋅ Mengdi Wang ⋅ Tuo Zhao ⋅ Yu-Xiang Wang
Large Learning Rates Improve Generalization: But How Large Are We Talking About?
Ekaterina Lobacheva ⋅ Eduard Pokonechny ⋅ Maxim Kodryan ⋅ Dmitry Vetrov
Generalization Guarantees of Deep ResNets in the Mean-Field Regime
Yihang Chen ⋅ Fanghui Liu ⋅ Yiping Lu ⋅ Grigorios Chrysos ⋅ Volkan Cevher
Theoretical Explanation for Generalization from Adversarial Perturbations
Soichiro Kumano ⋅ Hiroshi Kera ⋅ Toshihiko Yamasaki
In-Context Convergence of Transformers
Yu Huang ⋅ Yuan Cheng ⋅ Yingbin Liang
How Two-Layer Neural Networks Learn, One (Giant) Step at a Time
Yatin Dandi ⋅ Florent Krzakala ⋅ Bruno Loureiro ⋅ Luca Pesce ⋅ Ludovic Stephan
Transformers as Support Vector Machines
Davoud Ataee Tarzanagh ⋅ Yingcong Li ⋅ Christos Thrampoulidis ⋅ Samet Oymak
Symmetric Mean-field Langevin Dynamics for Distributional Minimax Problems
Juno Kim ⋅ Kakei Yamamoto ⋅ Kazusato Oko ⋅ Zhuoran Yang ⋅ Taiji Suzuki
A Theoretical Study of Dataset Distillation
Zachary Izzo ⋅ James Zou
Introducing an Improved Information-Theoretic Measure of Predictive Uncertainty
Kajetan Schweighofer ⋅ Lukas Aichberger ⋅ Mykyta Ielanskyi ⋅ Sepp Hochreiter
Denoising Low-Rank Data Under Distribution Shift: Double Descent and Data Augmentation
Chinmaya Kausik ⋅ Kashvi Srivastava ⋅ Rishi Sonthalia
A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks
Behrad Moniri ⋅ Donghwan Lee ⋅ Hamed Hassani ⋅ Edgar Dobriban
Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data
Zhiwei Xu ⋅ Yutong Wang ⋅ Spencer Frei ⋅ Gal Vardi ⋅ Wei Hu
Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP
Zixiang Chen ⋅ Yihe Deng ⋅ Yuanzhi Li ⋅ Quanquan Gu
Provably Efficient CVaR RL in Low-rank MDPs
Yulai Zhao ⋅ Wenhao Zhan ⋅ Xiaoyan Hu ⋅ Ho-fung Leung ⋅ Farzan Farnia ⋅ Wen Sun ⋅ Jason Lee
Analysis of Task Transferability in Large Pre-trained Classifiers
Akshay Mehra ⋅ Yunbei Zhang ⋅ Jihun Hamm
On Scale-Invariant Sharpness Measures
Behrooz Tahmasebi ⋅ Ashkan Soleymani ⋅ Stefanie Jegelka ⋅ Patrick Jaillet
Gibbs-Based Information Criteria and the Over-Parameterized Regime
Haobo Chen ⋅ Yuheng Bu ⋅ Gregory Wornell
Grokking modular arithmetic can be explained by margin maximization
Mohamad Amin Mohamadi ⋅ Zhiyuan Li ⋅ Lei Wu ⋅ Danica J. Sutherland
Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: \\ Global Convergence Guarantees and Feature Learning
Fadhel Ayed ⋅ Francois Caron ⋅ Paul Jung ⋅ Juho Lee ⋅ Hoil Lee ⋅ Hongseok Yang
Flow-Based High-Dimensionally Distributional Robust Optimization
Chen Xu ⋅ Jonghyeok Lee ⋅ Xiuyuan Cheng ⋅ Yao Xie
How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations
Tianyu Guo ⋅ Wei Hu ⋅ Song Mei ⋅ Huan Wang ⋅ Caiming Xiong ⋅ Silvio Savarese ⋅ Yu Bai
A Theoretical Explanation of Deep RL Performance in Stochastic Environments
Cassidy Laidlaw ⋅ Banghua Zhu ⋅ Stuart J Russell ⋅ Anca Dragan
Continual Learning for Long-Tailed Recognition: Bridging the Gap in Theory and Practice
Mahdiyar Molahasani ⋅ Ali Etemad ⋅ Michael Greenspan
Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate
Miao Lu ⋅ Beining Wu ⋅ Xiaodong Yang ⋅ Difan Zou
Escaping Random Teacher Initialization Enhances Signal Propagation and Representations
Felix Sarnthein ⋅ Sidak Pal Singh ⋅ Antonio Orvieto ⋅ Thomas Hofmann
The Expressive Power of Transformers with Chain of Thought
William Merrill ⋅ Ashish Sabharwal
Transformers as Multi-Task Feature Selectors: Generalization Analysis of In-Context Learning
Hongkang Li ⋅ Meng Wang ⋅ Songtao Lu ⋅ Hui Wan ⋅ Xiaodong Cui ⋅ Pin-Yu Chen
MoXCo:How I learned to stop exploring and love my local minima?
Esha Singh ⋅ Shoham Sabach ⋅ Yu-Xiang Wang
First-order ANIL provably learns representations despite overparametrisation
Oguz Kaan Yuksel ⋅ Etienne Boursier ⋅ Nicolas Flammarion
A Data-Driven Measure of Relative Uncertainty for Misclassification Detection
Eduardo Dadalto Câmara Gomes ⋅ Marco Romanelli ⋅ Georg Pichler ⋅ Pablo Piantanida
Non-Vacuous Generalization Bounds for Large Language Models
Sanae Lotfi ⋅ Marc Finzi ⋅ Yilun Kuang ⋅ Tim G. J. Rudner ⋅ Micah Goldblum ⋅ Andrew Wilson
Learning from setbacks: the impact of adversarial initialization on generalization performance
Yatin Dandi ⋅ Stefani Karp ⋅ Francesca Mignacco ⋅ Kavya Ravichandran
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit
Blake Bordelon ⋅ Lorenzo Noci ⋅ Mufan Li ⋅ Boris Hanin ⋅ Cengiz Pehlevan
Estimating optimal PAC-Bayes bounds with Hamiltonian Monte Carlo
Szilvia Ujváry ⋅ Gergely Flamich ⋅ Vincent Fortuin ⋅ José Miguel Hernández-Lobato
Divergence at the Interpolation Threshold: Identifying, Interpreting \& Ablating the Sources of a Deep Learning Puzzle
Rylan Schaeffer ⋅ Zachary Robertson ⋅ Akhilan Boopathy ⋅ Mikail Khona ⋅ Ila Fiete ⋅ Andrey Gromov ⋅ Sanmi Koyejo
Adaptive Sharpness-Aware Pruning for Robust Sparse Networks
Anna Bair ⋅ Hongxu Yin ⋅ Maying Shen ⋅ Pavlo Molchanov ⋅ Jose M. Alvarez
Invariant Low-Dimensional Subspaces in Gradient Descent for Learning Deep Matrix Factorizations
Can Yaras ⋅ Peng Wang ⋅ Wei Hu ⋅ Zhihui Zhu ⋅ Laura Balzano ⋅ Qing Qu
How Structured Data Guides Feature Learning: A Case Study of the Parity Problem
Atsushi Nitanda ⋅ Kazusato Oko ⋅ Taiji Suzuki ⋅ Denny Wu
The Next Symbol Prediction Problem: PAC-learning and its relation to Language Models
Satwik Bhattamishra ⋅ Phil Blunsom ⋅ Varun Kanade
Why Do We Need Weight Decay for Overparameterized Deep Networks?
Maksym Andriushchenko ⋅ Francesco D'Angelo ⋅ Aditya Vardhan Varre ⋅ Nicolas Flammarion
The Double-Edged Sword: Perception and Uncertainty in Inverse Problems
Regev Cohen ⋅ Ehud Rivlin ⋅ Daniel Freedman
Near-Interpolators: Fast Norm Growth and Tempered Near-Overfitting
Yutong Wang ⋅ Rishi Sonthalia ⋅ Wei Hu
Are Graph Neural Networks Optimal Approximation Algorithms?
Morris Yau ⋅ Eric Lu ⋅ Nikolaos Karalias ⋅ Jessica Xu ⋅ Stefanie Jegelka
JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention
Yuandong Tian ⋅ Yiping Wang ⋅ Zhenyu Zhang ⋅ Beidi Chen ⋅ Simon Du