Skip to yearly menu bar Skip to main content


Workshop

Third Workshop on Efficient Natural Language and Speech Processing (ENLSP-III): Towards the Future of Large Language Models and their Emerging Descendants

Mehdi Rezagholizadeh ⋅ Peyman Passban ⋅ Yue Dong ⋅ Yu Cheng ⋅ Soheila Samiee ⋅ Lili Mou ⋅ Qun Liu ⋅ Boxing Chen
Project Page

Abstract

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles
6:15 AM
Breakfast
6:16 AM
Opening Speech
Mehdi Rezagholizadeh
6:45 AM
Simple and efficient self-training approaches for speech recognition
Tatiana Likhomanenko ⋅ Samy Bengio
7:30 AM
[Paper-Oral 1] Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL
Hao Sun ⋅ Alihan Hüyük ⋅ Mihaela van der Schaar
7:36 AM
[Paper-Oral 2] MatFormer: Nested Transformer for Elastic Inference
Fnu Devvrit ⋅ Sneha Kudugunta ⋅ Aditya Kusupati ⋅ Tim Dettmers ⋅ Kaifeng Chen ⋅ Inderjit Dhillon ⋅ Yulia Tsvetkov ⋅ Hanna Hajishirzi ⋅ Sham Kakade ⋅ Ali Farhadi ⋅ Prateek Jain
7:42 AM
[Paper-Oral 3] Decoding Data Quality via Synthetic Corruptions: Embedding-guided Pruning of Code Data
Yu Yang ⋅ Aaditya Singh ⋅ Mostafa Elhoushi ⋅ Anas Mahmoud ⋅ Kushal Tirumala ⋅ Fabian Gloeckle ⋅ Baptiste Roziere ⋅ Carole-Jean Wu ⋅ Ari Morcos ⋅ Newsha Ardalani
7:48 AM
[Paper-Oral 4] FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Cores
Dan Fu ⋅ Hermann Kumbong ⋅ Eric Nguyen ⋅ Christopher Ré
7:54 AM
[Paper-Oral 5] Ensemble of low-rank adapters for large language model fine-tuning
Xi Wang ⋅ Laurence Aitchison ⋅ Maja Rudolph
9:30 AM
[Paper-Oral 6] LoDA: Low-Dimensional Adaptation of Large Language Models
Jing Liu ⋅ Toshiaki Koike-Akino ⋅ Perry Wang ⋅ Matthew Brand ⋅ Ye Wang ⋅ Kieran Parsons
9:36 AM
[Paper-Oral 7] MultiPrompter: Cooperative Prompt Optimization with Multi-Agent Reinforcement Learning
Dong-Ki Kim ⋅ Sungryull Sohn ⋅ Lajanugen Logeswaran ⋅ Dongsub Shim ⋅ Honglak Lee
9:42 AM
[Paper-Oral 8] LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
Yixiao Li ⋅ Yifan Yu ⋅ Chen Liang ⋅ Nikos Karampatziakis ⋅ Pengcheng He ⋅ Weizhu Chen ⋅ Tuo Zhao
9:48 AM
[Paper-Oral 9] Improving Linear Attention via Softmax Mimicry
Michael Zhang ⋅ Kush Bhatia ⋅ Hermann Kumbong ⋅ Christopher Ré
9:54 AM
[Paper-Oral 10] PaSS: Parallel Speculative Sampling
Giovanni Monea ⋅ Armand Joulin ⋅ Edouard Grave
10:00 AM
Lunch Break
1:20 PM
Interactive Panel Discussion
Nazneen Rajani ⋅ Tim Dettmers ⋅ Minjia Zhang
2:10 PM
Best Paper and Poster Awards
Mehdi Rezagholizadeh
What is Lost in Knowledge Distillation?
Manas Ranjan Mohanty ⋅ Tanya Roosta ⋅ Peyman Passban
LLM-MQ: Mixed-precision Quantization for Efficient LLM Deployment
Shiyao Li ⋅ Xuefei Ning ⋅ Ke Hong ⋅ Tengxuan Liu ⋅ Luning Wang ⋅ Xiuhong Li ⋅ Kai Zhong ⋅ Guohao Dai ⋅ Huazhong Yang ⋅ Yu Wang
Transfer Learning for Structured Pruning under Limited Task Data
Lucio M Dery ⋅ Awni Hannun ⋅ David Grangier
Embedding User-Generated Content using Structural Supervision and Generative Models
Vinay Shukla ⋅ Yang Yang ⋅ Siddarth Malreddy ⋅ Jinoo Baek ⋅ Dale Johnson ⋅ Wenfei Zou ⋅ Karthik Lakshmanan ⋅ Mark Williams ⋅ Minh Pham
Parameter Efficient Finetuning for Reducing Activation Density in Transformers
Bharat Runwal ⋅ Tejaswini Pedapati ⋅ Pin-Yu Chen
GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values
Farnoosh Javadi ⋅ Walid Ahmed ⋅ Habib Hajimolahoseini ⋅ Foozhan Ataiefard ⋅ Mohammad Hassanpour ⋅ Saina Asani ⋅ Austin Wen ⋅ Omar Mohamed Awad ⋅ Kangling Liu ⋅ Yang Liu
Structure Discovery in Prompted Weak Supervision
Jinyan Su ⋅ Peilin Yu ⋅ Jieyu Zhang ⋅ Stephen Bach
SPEED: Speculative Pipelined Execution for Efficient Decoding
Coleman Hooper ⋅ Sehoon Kim ⋅ Hiva Mohammadzadeh ⋅ Hasan Genc ⋅ Kurt Keutzer ⋅ Amir Gholami ⋅ Sophia Shao
Efficiently Adapting Pretrained Language Models to New Languages
Zoltan Csaki ⋅ Pian Pawakapan ⋅ Urmish Thakker ⋅ Qiantong Xu
Efficient LLM Inference on CPUs
Haihao Shen ⋅ Hanwen Chang ⋅ Bo Dong ⋅ Hengyu Meng ⋅ Yu Luo
Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer
Qingru Zhang ⋅ Dhananjay Ram ⋅ Cole Hawkins ⋅ Sheng Zha ⋅ Tuo Zhao
On the Zero-Shot Generalization of Machine-Generated Text Detectors
Xiao Pu ⋅ Jingyu Zhang ⋅ Xiaochuang Han ⋅ Yulia Tsvetkov ⋅ Tianxing He
Intra-Class Similarity-Guided Feature Distillation
Khouloud Saadi ⋅ Jelena Mitrović ⋅ Michael Granitzer
Lightweight Retrieval Tuning for Black-Box Language Models
Xiao-Wen Yang ⋅ Hong-Jie You ⋅ Pengxiao Song ⋅ Hao-Ran Hao ⋅ Jie-Jing Shao ⋅ Yu-Feng Li
Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding
Xuefei Ning ⋅ Zinan Lin ⋅ Zixuan Zhou ⋅ Zifu Wang ⋅ Huazhong Yang ⋅ Yu Wang
Investigating the Impact of Compression on Parametric Knowledge in Language Models
Satya Sai Srinath Namburi ⋅ Makesh Narsimhan Sreedhar ⋅ Srinath Srinivasan ⋅ Frederic Sala
Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs
Feiyang Kang ⋅ Hoang Anh Just ⋅ Himanshu Jahagirdar ⋅ Yifan Sun ⋅ Yuanzhi Zhang ⋅ Rongxing Du ⋅ Anit Kumar Sahu ⋅ Ruoxi Jia
Exploiting Transformer Activation Sparsity with Dynamic Inference
Mikołaj Piórczyński ⋅ Filip Szatkowski ⋅ Klaudia Bałazy ⋅ Bartosz Wójcik
Retrieval Augmented Generation for Dialog Modeling
Lilly Kumari ⋅ Usama Bin Shafqat ⋅ Nikhil Sarda
TCNCA: Temporal Convolution Network with Chunked Attention for Scalable Sequence Processing
Aleksandar Terzic ⋅ Michael Hersche ⋅ Geethan Karunaratne ⋅ Luca Benini ⋅ Abu Sebastian ⋅ Abbas Rahimi
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference Using Sorted Fine-Tuning (SoFT)
Parsa Kavehzadeh ⋅ Mojtaba Valipour ⋅ Marzieh Tahaei ⋅ Ali Ghodsi ⋅ Boxing Chen ⋅ Mehdi Rezaghoizadeh
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
Mengzhou Xia ⋅ Tianyu Gao ⋅ Zhiyuan Zeng ⋅ Danqi Chen
BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model
Nolan Dey ⋅ Daria Soboleva ⋅ Faisal Al-Khateeb ⋅ Bowen Yang ⋅ Ribhu Pathria ⋅ Hemant Khachane ⋅ Shaheer Muhammad ⋅ Zhiming (Charles) Chen ⋅ Robert Myers ⋅ Jacob Robert Steeves ⋅ Natalia Vassilieva ⋅ Marvin Tom ⋅ Joel Hestness
Sparse Fine-Tuning for Inference Acceleration of Large Language Models
Eldar Kurtic ⋅ Denis Kuznedelev ⋅ Elias Frantar ⋅ Michael Goin ⋅ Dan Alistarh
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
Suyu Ge ⋅ Yunan Zhang ⋅ Liyuan Liu ⋅ Minjia Zhang ⋅ Jiawei Han ⋅ Jianfeng Gao
MUX-PLMs: Data Multiplexing for High-throughput Language Models
Vishvak Murahari ⋅ Ameet Deshpande ⋅ Carlos Jimenez ⋅ Izhak Shafran ⋅ Mingqiu Wang ⋅ Yuan Cao ⋅ Karthik Narasimhan
Towards End-to-end 4-Bit Inference on Generative Large Language Models
Saleh Ashkboos ⋅ Ilia Markov ⋅ Elias Frantar ⋅ Tingxuan Zhong ⋅ Xincheng Wang ⋅ Jie Ren ⋅ Torsten Hoefler ⋅ Dan Alistarh
SortedNet, a Place for Every Network and Every Network in its Place
Mojtaba Valipour ⋅ Mehdi Rezaghoizadeh ⋅ Hossein Rajabzadeh ⋅ Marzieh Tahaei ⋅ Boxing Chen ⋅ Ali Ghodsi
FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
Young Jin Kim ⋅ Rawn Henry ⋅ Raffy Fahim ⋅ Hany Awadalla
KronA: Parameter Efficient Tuning with Kronecker Adapter
Ali Edalati ⋅ Marzieh Tahaei ⋅ Ivan Kobyzev ⋅ Vahid Partovi Nia ⋅ James J. Clark ⋅ Mehdi Rezaghoizadeh
ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
Seyed Iman Mirzadeh ⋅ Keivan Alizadeh-Vahid ⋅ Sachin Mehta ⋅ Carlo C Del Mundo ⋅ Oncel Tuzel ⋅ Golnoosh Samei ⋅ Mohammad Rastegari ⋅ Mehrdad Farajtabar
SwiftLearn: A Data-Efficient Training Method of Deep Learning Models using Importance Sampling
Habib Hajimolahoseini ⋅ Omar Mohamed Awad ⋅ Walid Ahmed ⋅ Austin Wen ⋅ Saina Asani ⋅ Mohammad Hassanpour ⋅ Farnoosh Javadi ⋅ Mehdi Ahmadi ⋅ Foozhan Ataiefard ⋅ Kangling Liu ⋅ Yang Liu
Efficient Stagewise Pretraining via Progressive Subnetworks
Abhishek Panigrahi ⋅ Nikunj Saunshi ⋅ Kaifeng Lyu ⋅ Sobhan Miryoosefi ⋅ Sashank Reddi ⋅ Satyen Kale ⋅ Sanjiv Kumar
Efficient Online Data Mixing For Language Model Pre-Training
Alon Albalak ⋅ Liang-Ming Pan ⋅ Colin Raffel ⋅ William Yang Wang
UT5: Pretraining Non autoregressive T5 with unrolled denoising
Mahmoud Salem ⋅ Jiayu Ye ⋅ Frederick Liu ⋅ Chu-Cheng Lin
LatticeGen: A Cooperative Framework Which Hides Generated Text in A Lattice For Privacy-Aware Generation on Cloud
Zhang ⋅ Tianxing He ⋅ Tianle Wang ⋅ Lu Mi ⋅ Niloofar Mireshghallah ⋅ Binyi Chen ⋅ Hao Wang ⋅ Yulia Tsvetkov
Measuring and Improving Recall in Convolutional Language Models
Evan Sabri Eyuboglu ⋅ Simran Arora ⋅ Aman Timalsina ⋅ Isys Johnson ⋅ Michael Poli ⋅ James Zou ⋅ Atri Rudra ⋅ Christopher Ré
Continual Pre-Training of Large Language Models: How to (re)warm your model?
Kshitij Gupta ⋅ Benjamin Thérien ⋅ Adam Ibrahim ⋅ Mats L Richter ⋅ Quentin Anthony ⋅ Eugene Belilovsky ⋅ Irina Rish ⋅ Timothee Lesort
Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion
Shangyu Wu ⋅ Ying Xiong ⋅ Yufei CUI ⋅ Xue (Steve) Liu ⋅ Buzhou Tang ⋅ Tei-Wei Kuo ⋅ Chun Jason XUE
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
Hossein Rajabzadeh ⋅ Mojtaba Valipour ⋅ Marzieh Tahaei ⋅ HYOCK JU KWON ⋅ Ali Ghodsi ⋅ Boxing Chen ⋅ Mehdi Rezaghoizadeh
Model Fusion through Bayesian Optimization in Language Model Fine-Tuning
Chaeyun Jang ⋅ Jungtaek Kim ⋅ Hyungi Lee ⋅ Juho Lee
Fast-ELECTRA for Efficient Pre-training
Chengyu Dong ⋅ Liyuan Liu ⋅ Hao Cheng ⋅ Jingbo Shang ⋅ Jianfeng Gao ⋅ Xiaodong Liu
Parameter-Efficient Fine-tuning of InstructBLIP for Visual Reasoning Tasks
Sungkyung Kim ⋅ Adam Lee ⋅ Junyoung Park ⋅ Sounho Chung ⋅ Jusang Oh ⋅ Jay Yoon Lee
Local LoRA: Memory-Efficient Fine-Tuning of Large Language Models
Oscar Key ⋅ Jean Kaddour ⋅ Pasquale Minervini
Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation
Zhewei Yao ⋅ Xiaoxia Wu ⋅ Cheng Li ⋅ Stephen Youn ⋅ Yuxiong He
DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
Conglong Li ⋅ Zhewei Yao ⋅ Xiaoxia Wu ⋅ Minjia Zhang ⋅ Connor Holmes ⋅ Cheng Li ⋅ Yuxiong He
Arabic Mini-ClimateGPT : A Climate Change and Sustainability Tailored Arabic LLM
Sahal Shaji Mullappilly ⋅ Abdelrahman Shaker ⋅ Omkar Thawakar ⋅ Hisham Cholakkal ⋅ Rao Anwer ⋅ Salman Khan ⋅ Fahad Shahbaz
Multimodal Data and Resource Efficient Device-directed Speech Detection with Large Foundation Models
Dominik Wagner ⋅ Alexander Churchill ⋅ Siddharth Sigtia ⋅ Panayiotis Georgiou ⋅ Matt Mirsamadi ⋅ Aarshee Mishra ⋅ Erik Marchi
ASR Data Selection from Multiple Sources: A Practical Approach on Performance Scaling
Hoang Anh Just ⋅ I-Fan Chen ⋅ Feiyang Kang ⋅ Yuanzhi Zhang ⋅ Anit Kumar Sahu ⋅ Ruoxi Jia
Fed-EE: Federating Heterogeneous ASR Models using Early-Exit Architectures
Mohamed Nabih Ali Mohamed Nawar ⋅ Alessio Brutti ⋅ Falavigna Daniele
Efficient infusion of self-supervised representations in Automatic Speech Recognition
Darshan Prabhu ⋅ Sai Ganesh Mirishkar ⋅ Pankaj Wasnik
An efficient clustering algorithm for self-supervised speaker recognition
Abderrahim Fathan ⋅ Xiaolin Zhu ⋅ JAHANGIR ALAM
HateXplain Space Model: Fusing Robustness with Explainability in Hate Speech Analysis
Md Fahim ⋅ Md Shihab Shahriar ⋅ Mohammad Ruhul Amin