Skip to yearly menu bar Skip to main content

Workshop: Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023 (FL@FM-NeurIPS'23)

SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models

Sara Babakniya · Ahmed Elkordy · Yahya Ezzeldin · Qingfeng Liu · Kee-Bong Song · Mostafa El-Khamy · Salman Avestimehr

Keywords: [ Parameter Efficient Fine Tuning ] [ efficiency ] [ large language model ]

[ ] [ Project Page ]
Sat 16 Dec 12:30 p.m. PST — 12:40 p.m. PST

Abstract: Fine-tuning pre-trained models has gained significant success in delivering SOTA results across various NLP tasks. In the absence of centralized data, Federated Learning (FL) helps the model to benefit from clients' private data for fine-tuning. However, due to the limited communication, computation, and storage capabilities of edge devices and the huge sizes of popular pre-trained models, efficient fine-tuning is crucial. This work explores the opportunities and challenges of applying parameter efficient fine-tuning (PEFT) methods in FL for language tasks. Specifically, our investigations reveal that with increasing data heterogeneity across users, the gap between fully fine-tuning the model and employing PEFT methods widens. To bridge this performance gap, we propose a method, SLoRA, which overcomes the key limitations of LoRA in high heterogeneous data scenarios through a novel data-driven initialization technique. Our experimental results demonstrate that SLoRA achieves performance comparable to full fine-tuning, with significant sparse updates with $\sim 1\%$ density while reducing training time by up to $90\%$.

Chat is not available.