Skip to yearly menu bar Skip to main content


Workshop

Mathematics of Modern Machine Learning (M3L)

Bingbin Liu ⋅ Kaifeng Lyu ⋅ Sadhika Malladi ⋅ Samory Kpotufe ⋅ Stefanie Jegelka ⋅ Tengyu Ma ⋅ Zhiyuan Li
Project Page [ OpenReview] 

Abstract

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles
8:50 AM
Opening Remarks
10:30 AM
Classifier-Free Guidance is a Predictor-Corrector
Arwen Bradley ⋅ Preetum Nakkiran
10:45 AM
Towards characterizing the value of edge embeddings in Graph Neural Networks
Dhruv Rohatgi ⋅ Tanya Marwah ⋅ Zachary Lipton ⋅ Jianfeng Lu ⋅ Ankur Moitra ⋅ Andrej Risteski
11:00 AM
Can Neural Networks Achieve Optimal Computational-statistical Tradeoff? An Analysis on Single-Index Model
Siyu Chen ⋅ Beining Wu ⋅ Miao Lu ⋅ Zhuoran Yang ⋅ Tianhao Wang
11:15 AM
Poster Session 1
12:15 PM
Lunch Break
3:00 PM
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
Riccardo Grazzi ⋅ Julien Siems ⋅ Jörg Franke ⋅ Arber Zela ⋅ Frank Hutter ⋅ Massimiliano Pontil
3:15 PM
Understanding Factual Recall in Transformers via Associative Memories
Eshaan Nichani ⋅ Jason Lee ⋅ Alberto Bietti
3:30 PM
Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs
Tianyu Guo ⋅ Druv Pai ⋅ Yu Bai ⋅ Jiantao Jiao ⋅ Michael Jordan ⋅ Song Mei
3:45 PM
Mixture of Parrots: Mixtures of experts improve memorization more than reasoning
Samy Jelassi ⋅ Clara Mohri ⋅ David Brandfonbrener ⋅ Alex Gu ⋅ Nikhil Vyas ⋅ Nikhil Anand ⋅ David Alvarez-Melis ⋅ Yuanzhi Li ⋅ Sham Kakade ⋅ Eran Malach
Self-Improvement in Language Models: The Sharpening Mechanism
Audrey Huang ⋅ Adam Block ⋅ Dylan J Foster ⋅ Dhruv Rohatgi ⋅ Cyril Zhang ⋅ Max Simchowitz ⋅ Jordan Ash ⋅ Akshay Krishnamurthy
SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network
Tomer Galanti ⋅ Zachary Siegel ⋅ Aparna Gupte ⋅ Tomaso Poggio
Emergence in non-neural models: grokking modular arithmetic via average gradient outer product
Neil Mallinar ⋅ Daniel Beaglehole ⋅ Libin Zhu ⋅ Adityanarayanan Radhakrishnan ⋅ Parthe Pandit ⋅ Misha Belkin
Diffusion Model Learns Low-Dimensional Distributions via Subspace Clustering
Peng Wang ⋅ Huijie Zhang ⋅ Zekai Zhang ⋅ Siyi Chen ⋅ Yi Ma ⋅ Qing Qu
Optimality and Adaptivity of Deep Neural Features for Instrumental Variable Regression
Juno Kim ⋅ Dimitri Meunier ⋅ Arthur Gretton ⋅ Taiji Suzuki ⋅ Zhu Li
Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection
Aaron Alvarado Kristanto Julistiono ⋅ Davoud Ataee Tarzanagh ⋅ Navid Azizan
Progressive distillation induces an implicit curriculum
Abhishek Panigrahi ⋅ Bingbin Liu ⋅ Sadhika Malladi ⋅ Andrej Risteski ⋅ Surbhi Goel
Comparing Implicit and Denoising Score-Matching Objectives
Artem Artemev ⋅ Ayan Das ⋅ Farhang Nabiei ⋅ Alberto Bernacchia
Understanding Diffusion-based Representation Learning via Low-Dimensional Modeling
Xiao Li ⋅ Zekai Zhang ⋅ Xiang Li ⋅ Siyi Chen ⋅ Zhihui Zhu ⋅ Peng Wang ⋅ Qing Qu
Benign Overfitting in Single-Head Attention
Roey Magen ⋅ Shuning Shang ⋅ Zhiwei Xu ⋅ Spencer Frei ⋅ Wei Hu ⋅ Gal Vardi
The GAN is dead; long live the GAN! A Modern GAN Baseline
Nick Huang ⋅ Aaron Gokaslan ⋅ Volodymyr Kuleshov ⋅ James Tompkin
Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training
Anchit Jain ⋅ Rozhin Nobahari ⋅ Aristide Baratin ⋅ Stefano Sarao Mannelli
Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models
Frederik Kunstner ⋅ Robin Yadav ⋅ Alan Milligan ⋅ Mark Schmidt ⋅ Alberto Bietti
On Your Mark, Get Set, Warmup!
Dayal Singh Kalra ⋅ Maissam Barkeshli
Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning
Alexey Rukhovich ⋅ Alexander Podolskiy ⋅ Irina Piontkovskaya
Transformers are Efficient Compilers, Provably
Xiyu Zhai ⋅ Runlong Zhou ⋅ Liao Zhang ⋅ Simon Du
Towards the Effect of Examples on In-Context Learning: A Theoretical Case Study
Pengfei He ⋅ Yingqian Cui ⋅ Han Xu ⋅ Hui Liu ⋅ Makoto Yamada ⋅ Jiliang Tang ⋅ Yue XING
Towards characterizing the value of edge embeddings in Graph Neural Networks
Dhruv Rohatgi ⋅ Tanya Marwah ⋅ Zachary Lipton ⋅ Jianfeng Lu ⋅ Ankur Moitra ⋅ Andrej Risteski
Benign Overfitting in Out-of-Distribution Generalization of Linear Models
Shange Tang ⋅ Jiayun Wu ⋅ Jianqing Fan ⋅ Chi Jin
Dynamics of Concept Learning and Compositional Generalization
Yongyi Yang ⋅ Core Francisco Park ⋅ Ekdeep S Lubana ⋅ Maya Okawa ⋅ Wei Hu ⋅ Hidenori Tanaka
Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs
Jonas Hübotter ⋅ Sascha Bongni ⋅ Ido Hakimi ⋅ Andreas Krause
Declarative characterizations of direct preference alignment algorithms
Kyle Richardson ⋅ Vivek Srikumar ⋅ Ashish Sabharwal
From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks
Clémentine Dominé ⋅ Nicolas Anguita ⋅ Alexandra Proca ⋅ Lukas Braun ⋅ Daniel Kunin ⋅ Pedro A.M Mediano ⋅ Andrew Saxe
Sample compression unleashed : New generalization bounds for real valued losses
Mathieu Bazinet ⋅ Valentina Zantedeschi ⋅ Pascal Germain
Simple and Effective Masked Diffusion Language Models
Subham Sahoo ⋅ Marianne Arriola ⋅ Aaron Gokaslan ⋅ Yair Schiff ⋅ Edgar Marroquin ⋅ Justin Chiu ⋅ Alexander Rush ⋅ Volodymyr Kuleshov
Understanding Factual Recall in Transformers via Associative Memories
Eshaan Nichani ⋅ Jason Lee ⋅ Alberto Bietti
A Theory of Initialisation's Impact on Specialisation
Devon Jarvis ⋅ Sebastian Lee ⋅ Clémentine Dominé ⋅ Andrew Saxe ⋅ Stefano Sarao Mannelli
Diffusion Models With Learned Adaptive Noise Processes
Subham Sahoo ⋅ Aaron Gokaslan ⋅ Christopher De Sa ⋅ Volodymyr Kuleshov
Harnessing the Power of Vicinity-Informed Analysis for Classification under Covariate Shift
Mitsuhiro Fujikawa ⋅ Youhei Akimoto ⋅ Jun Sakuma ⋅ Kazuto Fukuchi
A Theoretical Framework for Federated Domain Generalization with Gradient Alignment
Mahdiyar Molahasani ⋅ Milad Soltany ⋅ Farhad Pourpanah ⋅ Michael Greenspan ⋅ Ali Etemad
Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs
Tianyu Guo ⋅ Druv Pai ⋅ Yu Bai ⋅ Jiantao Jiao ⋅ Michael Jordan ⋅ Song Mei
In-Context Learning by Linear Attention: Exact Asymptotics and Experiments
Yue Lu ⋅ Mary Letey ⋅ Jacob Zavatone-Veth ⋅ Anindita Maiti ⋅ Cengiz Pehlevan
Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
Yuda Song ⋅ Hanlin Zhang ⋅ Udaya Ghai ⋅ Carson Eisenach ⋅ Sham Kakade ⋅ Dean Foster
Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
Riccardo Grazzi ⋅ Julien Siems ⋅ Jörg Franke ⋅ Arber Zela ⋅ Frank Hutter ⋅ Massimiliano Pontil
A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules
Kairong Luo ⋅ Haodong Wen ⋅ Shengding Hu ⋅ Zhenbo Sun ⋅ Zhiyuan Liu ⋅ Maosong Sun ⋅ Kaifeng Lyu ⋅ Wenguang Chen
Algorithmic Stability of Minimum-Norm Interpolating Deep Neural Networks
Ouns El Harzli ⋅ yoonsoo nam ⋅ Ilja Kuzborskij ⋅ Bernardo Grau ⋅ Ard Louis
Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
Noam Razin ⋅ Sadhika Malladi ⋅ Adithya Bhaskar ⋅ Danqi Chen ⋅ Sanjeev Arora ⋅ Boris Hanin
Provable unlearning in topic modeling and downstream tasks
Stanley Wei ⋅ Sadhika Malladi ⋅ Sanjeev Arora ⋅ Amartya Sanyal
Implicit Bias of Adam versus Gradient Descent in One-Hidden-Layer Neural Networks
Bhavya Vasudeva ⋅ Vatsal Sharan ⋅ Mahdi Soltanolkotabi
HERTA: A High-Efficiency and Rigorous Training Algorithm for Unfolded Graph Neural Networks
Yongyi Yang ⋅ Jiaming Yang ⋅ Wei Hu ⋅ Michal Derezinski
Parameter Symmetry and Emergence of Noise Equilibrium in Stochastic Training
Liu Ziyin ⋅ Mingze Wang ⋅ Hongchao Li ⋅ Lei Wu
Mixture of Parrots: Mixtures of experts improve memorization more than reasoning
Samy Jelassi ⋅ Clara Mohri ⋅ David Brandfonbrener ⋅ Alex Gu ⋅ Nikhil Vyas ⋅ Nikhil Anand ⋅ David Alvarez-Melis ⋅ Yuanzhi Li ⋅ Sham Kakade ⋅ Eran Malach
Can Neural Networks Achieve Optimal Computational-statistical Tradeoff? An Analysis on Single-Index Model
Siyu Chen ⋅ Beining Wu ⋅ Miao Lu ⋅ Zhuoran Yang ⋅ Tianhao Wang
Label Noise: Ignorance Is Bliss
Yilun Zhu ⋅ Jianxin Zhang ⋅ Aditya Gangrade ⋅ Clay Scott
Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
Francesco Mori ⋅ Stefano Sarao Mannelli ⋅ Francesca Mignacco
Accumulating Data Avoids Model Collapse
Joshua Kazdan ⋅ Apratim Dey ⋅ Rylan Schaeffer ⋅ Matthias Gerstgrasser ⋅ Rafael Rafailov ⋅ David Donoho ⋅ Sanmi Koyejo
Robust Feature Learning for Multi-Index Models in High Dimensions
Alireza Mousavi-Hosseini ⋅ Adel Javanmard ⋅ Murat Erdogdu
Classifier-Free Guidance is a Predictor-Corrector
Arwen Bradley ⋅ Preetum Nakkiran
Towards Principled Graph Transformers
Luis Müller ⋅ Daniel Kusuma ⋅ Blai Bonet ⋅ Christopher Morris