Workshop

Mathematics of Modern Machine Learning (M3L)

Zhiyuan Li · Tengyu Ma · Surbhi Goel · Kaifeng Lyu · Christina Baek · Bingbin Liu · Alex Damian · Aditi Raghunathan

Project Page

Abstract

This workshop explores theory for understanding and advancing modern ML practices: optimization, generalization, and foundation models.

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles

6:50 AM

Opening Remarks Opening Remarks

Video

7:00 AM

From algorithms to neural networks and back Invited Talk

Andrej Risteski

Video

7:45 AM

How do two-layer neural networks learn complex functions from data over time? Invited Talk

Florent Krzakala

Video

8:30 AM

Feature Learning in Infinite-Depth Neural Networks Oral

Greg Yang · Dingli Yu · Chen Zhu · Soufiane Hayou

Video

Link

8:40 AM

Fit Like You Sample: Sample-Efficient Score Matching From Fast Mixing Diffusions Oral

Yilong Qin · Andrej Risteski

Video

Link

8:50 AM

Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models Oral

Song Mei · Yuchen Wu

Video

Link

9:00 AM

Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data Oral

Zhiwei Xu · Yutong Wang · Spencer Frei · Gal Vardi · Wei Hu

Video

Link

9:10 AM

Poster Session Poster Session

10:10 AM

Lunch Break Lunch Break

11:15 AM

Benefits of learning with symmetries: eigenvectors, graph representations and sample complexity Invited Talk

Stefanie Jegelka

Video

12:00 PM

Break

12:15 PM

Adaptivity in Domain Adaptation and Friends Invited Talk

Samory Kpotufe

Video

1:00 PM

Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit Oral

Blake Bordelon · Lorenzo Noci · Mufan Li · Boris Hanin · Cengiz Pehlevan

Video

Link

1:10 PM

Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP Oral

Zixiang Chen · Yihe Deng · Yuanzhi Li · Quanquan Gu

Video

Link

1:20 PM

In-Context Convergence of Transformers Oral

Yu Huang · Yuan Cheng · Yingbin Liang

Video

Link

1:30 PM

Large Catapults in Momentum Gradient Descent with Warmup: An Empirical Study Oral

Prin Phunyaphibarn · Junghyun Lee · Bohan Wang · Huishuai Zhang · Chulhee Yun

Video

Link

1:40 PM

Linear attention is (maybe) all you need (to understand transformer optimization) Oral

Kwangjun Ahn · Xiang Cheng · Minhak Song · Chulhee Yun · Ali Jadbabaie · Suvrit Sra

Video

Link

1:50 PM

Closing Remarks Closing Remarks

Video

2:00 PM

Poster Session Poster Session

A PAC-Bayesian Perspective on the Interpolating Information Criterion Poster

Liam Hodgkinson · Chris van der Heide · Robert Salomone · Fred Roosta · Michael Mahoney

Link

Graph Neural Networks Benefit from Structural Information Provably: A Feature Learning Perspective Poster

Wei Huang · Yuan Cao · Haonan Wang · Xin Cao · Taiji Suzuki

Link

Linear attention is (maybe) all you need (to understand transformer optimization) Poster

Kwangjun Ahn · Xiang Cheng · Minhak Song · Chulhee Yun · Ali Jadbabaie · Suvrit Sra

Link

Large Catapults in Momentum Gradient Descent with Warmup: An Empirical Study Poster

Prin Phunyaphibarn · Junghyun Lee · Bohan Wang · Huishuai Zhang · Chulhee Yun

Link

Feature Learning in Infinite-Depth Neural Networks Poster

Greg Yang · Dingli Yu · Chen Zhu · Soufiane Hayou

Link

Variational Classification Poster

Shehzaad Dhuliawala · Mrinmaya Sachan · Carl Allen

Link

Implicit biases in multitask and continual learningfrom a backward error analysis perspective Poster

Benoit Dherin

Link

Spectrum Extraction and Clipping for Implicitly Linear Layers Poster

Ali Ebrahimpour-Boroojeny · Matus Telgarsky · Hari Sundaram

Link

The Noise Geometry of Stochastic Gradient Descent: A Quantitative and Analytical Characterization Poster

Mingze Wang · Lei Wu

Link

Curvature-Dimension Tradeoff for Generalization in Hyperbolic Space Poster

Nicolás Alvarado · Hans Lobel · Mircea Petrache

Link

Complexity Matters: Dynamics of Feature Learning in the Presence of Spurious Correlations Poster

GuanWen Qiu · Da Kuang · Surbhi Goel

Link

Unveiling the Hessian's Connection to the Decision Boundary Poster

Mahalakshmi Sabanayagam · Freya Behrens · Urte Adomaityte · Anna Dawid

Link

Nonparametric Classification on Low Dimensional Manifolds using Overparameterized Convolutional Residual Networks Poster

Zixuan Zhang · Kaiqi Zhang · Minshuo Chen · Yuma Takeda · Mengdi Wang · Tuo Zhao · Yu-Xiang Wang

Link

Large Learning Rates Improve Generalization: But How Large Are We Talking About? Poster

Ekaterina Lobacheva · Eduard Pokonechny · Maxim Kodryan · Dmitry Vetrov

Link

Understanding the Role of Noisy Statistics in the Regularization Effect of Batch Normalization Poster

Atli Kosson · Dongyang Fan · Martin Jaggi

Link

Generalization Guarantees of Deep ResNets in the Mean-Field Regime Poster

Yihang Chen · Fanghui Liu · Yiping Lu · Grigorios Chrysos · Volkan Cevher

Link

Theoretical Explanation for Generalization from Adversarial Perturbations Poster

Soichiro Kumano · Hiroshi Kera · Toshihiko Yamasaki

Link

In-Context Convergence of Transformers Poster

Yu Huang · Yuan Cheng · Yingbin Liang

Link

How Two-Layer Neural Networks Learn, One (Giant) Step at a Time Poster

Yatin Dandi · Florent Krzakala · Bruno Loureiro · Luca Pesce · Ludovic Stephan

Link

Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States Poster

Ziqiao Wang · Yongyi Mao

Link

Unraveling the Complexities of Simplicity Bias: Mitigating and Amplifying Factors Poster

Xuchen Gong · Tianwen Fu

Link

Transformers as Support Vector Machines Poster

Davoud Ataee Tarzanagh · Yingcong Li · Christos Thrampoulidis · Samet Oymak

Link

Symmetric Mean-field Langevin Dynamics for Distributional Minimax Problems Poster

Juno Kim · Kakei Yamamoto · Kazusato Oko · Zhuoran Yang · Taiji Suzuki

Link

A Theoretical Study of Dataset Distillation Poster

Zachary Izzo · James Zou

Link

Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models Poster

Deqing Fu · Tian-qi Chen · Robin Jia · Vatsal Sharan

Link

Introducing an Improved Information-Theoretic Measure of Predictive Uncertainty Poster

Kajetan Schweighofer · Lukas Aichberger · Mykyta Ielanskyi · Sepp Hochreiter

Link

In-Context Learning on Unstructured Data: Softmax Attention as a Mixture of Experts Poster

Kevin Christian Wibisono · Yixin Wang

Link

Attention-Only Transformers and Implementing MLPs with Attention Heads Poster

Robert Huben · Valerie Morris

Link

Privacy at Interpolation: Precise Analysis for Random and NTK Features Poster

Simone Bombari · Marco Mondelli

Link

Denoising Low-Rank Data Under Distribution Shift: Double Descent and Data Augmentation Poster

Chinmaya Kausik · Kashvi Srivastava · Rishi Sonthalia

Link

A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks Poster

Behrad Moniri · Donghwan Lee · Hamed Hassani · Edgar Dobriban

Link

Benign Overfitting and Grokking in ReLU Networks for XOR Cluster Data Poster

Zhiwei Xu · Yutong Wang · Spencer Frei · Gal Vardi · Wei Hu

Link

How does Gradient Descent Learn Features --- A Local Analysis for Regularized Two-Layer Neural Networks Poster

Mo Zhou · Rong Ge

Link

Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP Poster

Zixiang Chen · Yihe Deng · Yuanzhi Li · Quanquan Gu

Link

Provably Efficient CVaR RL in Low-rank MDPs Poster

Yulai Zhao · Wenhao Zhan · Xiaoyan Hu · Ho-fung Leung · Farzan Farnia · Wen Sun · Jason Lee

Link

Analysis of Task Transferability in Large Pre-trained Classifiers Poster

Akshay Mehra · Yunbei Zhang · Jihun Hamm

Link

On Scale-Invariant Sharpness Measures Poster

Behrooz Tahmasebi · Ashkan Soleymani · Stefanie Jegelka · Patrick Jaillet

Link

Gibbs-Based Information Criteria and the Over-Parameterized Regime Poster

Haobo Chen · Yuheng Bu · Gregory Wornell

Link

Grokking modular arithmetic can be explained by margin maximization Poster

Mohamad Amin Mohamadi · Zhiyuan Li · Lei Wu · Danica J. Sutherland

Link

Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: \\ Global Convergence Guarantees and Feature Learning Poster

Fadhel Ayed · Francois Caron · Paul Jung · Juho Lee · Hoil Lee · Hongseok Yang

Link

On the Computational Complexity of Inverting Generative Models Poster

Feyza Duman Keles · Chinmay Hegde

Link

Flow-Based High-Dimensionally Distributional Robust Optimization Poster

Chen Xu · Jonghyeok Lee · Xiuyuan Cheng · Yao Xie

Link

Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining Poster

Licong Lin · Yu Bai · Song Mei

Link

How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations Poster

Tianyu Guo · Wei Hu · Song Mei · Huan Wang · Caiming Xiong · Silvio Savarese · Yu Bai

Link

A Theoretical Explanation of Deep RL Performance in Stochastic Environments Poster

Cassidy Laidlaw · Banghua Zhu · Stuart J Russell · Anca Dragan

Link

Deep Networks as Denoising Algorithms: Sample-Efficient Learning of Diffusion Models in High-Dimensional Graphical Models Poster

Song Mei · Yuchen Wu

Link

Under-Parameterized Double Descent for Ridge Regularized Least Squares Denoising of Data on a Line Poster

Rishi Sonthalia · Xinyue (Serena) Li · Bochao Gu

Link

Continual Learning for Long-Tailed Recognition: Bridging the Gap in Theory and Practice Poster

Mahdiyar Molahasani · Ali Etemad · Michael Greenspan

Link

SimVAE: Narrowing the gap between Discriminative & Generative Representation Learning Poster

Alice Bizeul · Carl Allen

Link

Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks Poster

Atli Kosson · Bettina Messmer · Martin Jaggi

Link

Benign Oscillation of Stochastic Gradient Descent with Large Learning Rate Poster

Miao Lu · Beining Wu · Xiaodong Yang · Difan Zou

Link

On Compositionality and Emergence in Physical Systems Generativie Modeling Poster

Justin Diamond

Link

Escaping Random Teacher Initialization Enhances Signal Propagation and Representations Poster

Felix Sarnthein · Sidak Pal Singh · Antonio Orvieto · Thomas Hofmann

Link

The Expressive Power of Transformers with Chain of Thought Poster

William Merrill · Ashish Sabharwal

Link

Transformers as Multi-Task Feature Selectors: Generalization Analysis of In-Context Learning Poster

Hongkang Li · Meng Wang · Songtao Lu · Hui Wan · Xiaodong Cui · Pin-Yu Chen

Link

Fit Like You Sample: Sample-Efficient Score Matching From Fast Mixing Diffusions Poster

Yilong Qin · Andrej Risteski

Link

Towards the Fundamental Limits of Knowledge Transfer over Finite Domains Poster

Qingyue Zhao · Banghua Zhu

Link

Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization Poster

Elan Rosenfeld · Andrej Risteski

Link

MoXCo:How I learned to stop exploring and love my local minima? Poster

Esha Singh · Shoham Sabach · Yu-Xiang Wang

Link

First-order ANIL provably learns representations despite overparametrisation Poster

Oguz Kaan Yuksel · Etienne Boursier · Nicolas Flammarion

Link

A Data-Driven Measure of Relative Uncertainty for Misclassification Detection Poster

Eduardo Dadalto Câmara Gomes · Marco Romanelli · Georg Pichler · Pablo Piantanida

Link

Non-Vacuous Generalization Bounds for Large Language Models Poster

Sanae Lotfi · Marc Finzi · Yilun Kuang · Tim G. J. Rudner · Micah Goldblum · Andrew Wilson

Link

Learning from setbacks: the impact of adversarial initialization on generalization performance Poster

Yatin Dandi · Stefani Karp · Francesca Mignacco · Kavya Ravichandran

Link

Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit Poster

Blake Bordelon · Lorenzo Noci · Mufan Li · Boris Hanin · Cengiz Pehlevan

Link

Estimating optimal PAC-Bayes bounds with Hamiltonian Monte Carlo Poster

Szilvia Ujváry · Gergely Flamich · Vincent Fortuin · José Miguel Hernández-Lobato

Link

Divergence at the Interpolation Threshold: Identifying, Interpreting \& Ablating the Sources of a Deep Learning Puzzle Poster

Rylan Schaeffer · Zachary Robertson · Akhilan Boopathy · Mikail Khona · Ila Fiete · Andrey Gromov · Sanmi Koyejo

Link

Good regularity creates large learning rate implicit biases: edge of stability, balancing, and catapult Poster

Yuqing Wang · Zhenghao Xu · Tuo Zhao · Molei Tao

Link

Toward Student-oriented Teacher Network Training for Knowledge Distillation Poster

Chengyu Dong · Liyuan Liu · Jingbo Shang

Link

Adaptive Sharpness-Aware Pruning for Robust Sparse Networks Poster

Anna Bair · Hongxu Yin · Maying Shen · Pavlo Molchanov · Jose M. Alvarez

Link

Invariant Low-Dimensional Subspaces in Gradient Descent for Learning Deep Matrix Factorizations Poster

Can Yaras · Peng Wang · Wei Hu · Zhihui Zhu · Laura Balzano · Qing Qu

Link

How Structured Data Guides Feature Learning: A Case Study of the Parity Problem Poster

Atsushi Nitanda · Kazusato Oko · Taiji Suzuki · Denny Wu

Link

The Next Symbol Prediction Problem: PAC-learning and its relation to Language Models Poster

Satwik Bhattamishra · Phil Blunsom · Varun Kanade

Link

Why Do We Need Weight Decay for Overparameterized Deep Networks? Poster

Maksym Andriushchenko · Francesco D'Angelo · Aditya Vardhan Varre · Nicolas Flammarion

Link

The Double-Edged Sword: Perception and Uncertainty in Inverse Problems Poster

Regev Cohen · Ehud Rivlin · Daniel Freedman

Link

Near-Interpolators: Fast Norm Growth and Tempered Near-Overfitting Poster

Yutong Wang · Rishi Sonthalia · Wei Hu

Link

On robust overfitting: adversarial training induced distribution matters Poster

Runzhi Tian · Yongyi Mao

Link

Are Graph Neural Networks Optimal Approximation Algorithms? Poster

Morris Yau · Eric Lu · Nikolaos Karalias · Jessica Xu · Stefanie Jegelka

Link

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention Poster

Yuandong Tian · Yiping Wang · Zhenyu Zhang · Beidi Chen · Simon Du

Link