Skip to yearly menu bar Skip to main content


Workshop

Workshop on Advancing Neural Network Training (WANT): Computational Efficiency, Scalability, and Resource Optimization

Julia Gusak · Jean Kossaifi · Alena Shilova · Rocco Sedona · Cristiana Bentes · Animashree Anandkumar · Olivier Beaumont
Project Page

Abstract

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles
6:50 AM
Opening Remarks Talk
Julia Gusak
8:00 AM
Coffee Break
9:31 AM
Training and inference of large language models using 8-bit floating point Contributed Talk & Poster
Sergio Perez · Yan Zhang · James Briggs · Charles Blake · Josh Levy-Kramer · Paul Balanca · Carlo Luschi · Stephen Barlow · Andrew Fitzgibbon
9:37 AM
MatFormer: Nested Transformer for Elastic Inference Contributed Talk & Poster
Fnu Devvrit · Sneha Kudugunta · Aditya Kusupati · Tim Dettmers · Kaifeng Chen · Inderjit Dhillon · Yulia Tsvetkov · Hannaneh Hajishirzi · Sham Kakade · Ali Farhadi · Prateek Jain
9:43 AM
Sparse Backpropagation for MoE Training Contributed Talk & Poster
Liyuan Liu · Jianfeng Gao · Weizhu Chen
9:49 AM
Efficient Parallelization Layouts for Large-Scale Distributed Model Training Contributed Talk & Poster
Johannes Hagemann · Samuel Weinbach · Konstantin Dobler · Maximilian Schall · Gerard de Melo
9:55 AM
CoTFormer: More Tokens With Attention Make Up For Less Depth Contributed Talk & Poster
Amirkeivan Mohtashami · Matteo Pagliardini · Martin Jaggi
10:00 AM
Lunch
1:00 PM
Coffee Break
1:30 PM
Efficient LLM Training and Inference on GPUs Invited Talk
Mohammad Shoeybi · Bryan Catanzaro
2:00 PM
Panel Discussion Panel
Yang You · Olatunji Ruwase · Natalia Vassilieva · Mohammad Shoeybi · Ana Gainaru · Lionel Eyraud-Dubois · Jean Kossaifi
2:50 PM
Closing Remarks Talk
Jean Kossaifi
AI4HPC: Library to Train AI Models on HPC Systems using CFD Datasets Poster
Eray Inanc · Rakesh Sarma · Marcel Aach · Rocco Sedona · Andreas Lintermann
LightSeq: : Sequence Level Parallelism for Distributed Training of Long Context Transformers Poster
Dacheng Li · Rulin Shao · Anze Xie · Eric Xing · Joseph Gonzalez · Ion Stoica · Xuezhe Ma · Hao Zhang
FlexTrain: A Dynamic Training Framework for Heterogeneous Devices Environments Poster
Mert Unsal · Ali Maatouk · Antonio De Domenico · Nicola Piovesan · Fadhel Ayed
DYAD: A Descriptive Yet Abjuring Density efficient approximation to linear neural network layers Poster
Sarin Chandy · Varun Prashant Gangal · Yi Yang · Gabriel Maggiotti
Improving Deep Ensembles without Communication Poster
Konstantinos Pitas · Michael Arbel · Julyan Arbel
ConcatPlexer : Additional Dim1 Batching for Faster ViTs Contributed Talk & Poster
Donghoon Han · Seunghyeon Seo · Donghyeon Jeon · Jiho Jang · Chaerin Kong · Nojun Kwak
InstaTune: Instantaneous Neural Architecture Search During Fine-Tuning Poster
Sharath Nittur Sridhar · Souvik Kundu · Sairam Sundaresan · Maciej Szankin · Anthony Sarah
ReLoRA: High-Rank Training Through Low-Rank Updates Poster
Vladislav Lialin · Sherin Muckatira · Namrata Shivagunde · Anna Rumshisky
Sparse Iso-FLOP Transformations for Maximizing Training Efficiency Poster
Vithursan Thangarasa · Shreyas Saxena · Abhay Gupta · Sean Lie
Embarrassingly Simple Dataset Distillation Poster
Yunzhen Feng · Shanmukha Ramakrishna Vedantam · Julia Kempe
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs Poster
Suyu Ge · Yunan Zhang · Liyuan Liu · Minjia Zhang · Jiawei Han · Jianfeng Gao
A Quadratic Synchronization Rule for Distributed Deep Learning Poster
Xinran Gu · Kaifeng Lyu · Sanjeev Arora · Jingzhao Zhang · Longbo Huang
DAREL: Data Reduction with Losses for Training Acceleration of Real and Hypercomplex Neural Networks Poster
Alexander Demidovskij · Aleksei Trutnev · Artyom Tugaryov · Igor Salnikov · Stanislav Pavlov
Accelerating Deep Learning using Ivy Poster
Guillermo Sanchez-Brizuela · Ved Patwardhan · Matthew Barrett · Paul Anderson · Mustafa Hani · Daniel Lenton
LeanFlex-GKP: Advancing Hassle-Free Structured Pruning with Simple Flexible Group Count Poster
Jiamu Zhang · Shaochen (Henry) Zhong · Andrew Ye · Zirui Liu · Kaixiong Zhou · Xia Hu · Shuai Xu · Vipin Chaudhary
Patch Gradient Descent: Training Neural Networks on Very Large Images Poster
Deepak Gupta · Gowreesh Mago · Arnav Chavan · Dilip K. Prasad · Rajat Thomas
Local LoRA: Memory-Efficient Fine-Tuning of Large Language Models Poster
Oscar Key · Jean Kaddour · Pasquale Minervini
Early Weight Averaging meets High Learning Rates for LLM Pre-training Poster
Sunny Sanyal · Atula Neerkaje · Jean Kaddour · Abhishek Kumar · Sujay Sanghavi
Bandit-Driven Batch Selection for Robust Learning under Label Noise Poster
Michal Lisicki · Graham Taylor · Mihai Nica
Maestro: Uncovering Low-Rank Structures via Trainable Decomposition Poster
Samuel Horváth · Stefanos Laskaridis · Shashank Rajput · Hongyi Wang
DONUT-hole: DONUT Sparsification by Harnessing Knowledge and Optimizing Learning Efficiency Poster
azhar shaikh · Michael Cochez · Denis Diachkov · Michiel de Rijcke · Sahar Yousefi
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning Poster
Mengzhou Xia · Tianyu Gao · Zhiyuan Zeng · Danqi Chen
Task Arithmetic with LoRA for Continual Learning Poster
Rajas Chitale · Ankit Vaidya · Aditya Kane · Archana Ghotkar
Cooperative Learning for Cost-Adaptive Inference Poster
Xingli Fang · Richard Bradford · Jung-Eun Kim
Generalisable Agents for Neural Network Optimisation Poster
Kale-ab Tessera · Callum R. Tilbury · Sasha Abramowitz · Ruan John de Kock · Omayma Mahjoub · Benjamin Rosman · Sara Hooker · Arnu Pretorius