Timezone: »

 
Poster
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
Haokun Liu · Derek Tam · Mohammed Muqeeth · Jay Mohta · Tenghao Huang · Mohit Bansal · Colin Raffel

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #421
in Poster Session 2 »

Few-shot in-context learning (ICL) enables pre-trained language models to perform a previously-unseen task without any gradient-based training by feeding a small number of training examples as part of the input. ICL incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made. Parameter-efficient fine-tuning (PEFT) (e.g. adapter modules, prompt tuning, sparse update methods, etc.) offers an alternative paradigm where a small set of parameters are trained to enable a model to perform the new task. In this paper, we rigorously compare few-shot ICL and PEFT and demonstrate that the latter offers better accuracy as well as dramatically lower computational costs. Along the way, we introduce a new PEFT method called (IA)^3 that scales activations by learned vectors, attaining stronger performance while only introducing a relatively tiny amount of new parameters. We also propose a simple recipe based on the T0 model called T-Few that can be applied to new tasks without task-specific tuning or modifications. We validate the effectiveness of T-Few on completely unseen tasks by applying it to the RAFT benchmark, attaining super-human performance for the first time and outperforming the state-of-the-art by 6% absolute. All of the code used in our experiments will be publicly available.

Keywords: Few-Shot Learning · in-context learning · parameter-efficient training
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Haokun Liu (Department of Computer Science, University of North Carolina, Chapel Hill)
Derek Tam (Department of Computer Science, University of North Carolina, Chapel Hill)
Derek Tam

I am a Ph.D. student at UNC Chapel Hill working with Prof Colin Raffel and Prof Mohit Bansal . I am interested in updating lanuguage models efficiently and few-shot learning.

Mohammed Muqeeth (University of North Carolina at Chapel Hill)
Mohammed Muqeeth

I am interested in applying machine learning to solve NLP tasks efficiently

Jay Mohta (North Carolina State University)
Tenghao Huang (University of North Carolina, Chapel Hill)
Mohit Bansal (UNC Chapel Hill)
Colin Raffel (UNC Chapel Hill and Hugging Face)

More from the Same Authors