Timezone: »

 
Petals: Collaborative Inference and Fine-tuning of Large Models
Alexander Borzunov · Dmitry Baranchuk · Tim Dettmers · Max Ryabinin · Younes Belkada · Artem Chumachenko · Pavel Samygin · Colin Raffel

Sat Dec 03 08:30 AM -- 08:45 AM (PST) @
in Broadening Research Collaborations »
Event URL: https://openreview.net/forum?id=Ls_NTjgWXZV »
Many NLP tasks benefit from using large language models (LLMs) that often have more than 100 billion parameters. With the release of BLOOM-176B and OPT-175B, everyone can download pretrained models of this scale. Still, using these models requires high-end hardware unavailable to many researchers. In some cases, LLMs can be used more affordably via RAM offloading or hosted APIs. However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research. In this work, we propose Petals - a system for inference and fine-tuning of large models collaboratively by joining the resources of multiple parties. We demonstrate that this strategy significantly outperforms offloading for very large models, running inference of BLOOM-176B on consumer GPUs with $\approx$ 1 step per second. Unlike most inference APIs, Petals also natively exposes the hidden states of served models, allowing its users to train and share custom model extensions based on efficient fine-tuning methods.

Author Information

Alexander Borzunov (HSE University, Yandex)
Dmitry Baranchuk (MSU / Yandex)
Tim Dettmers (University of Washington)
Max Ryabinin (Yandex, HSE University)
Younes Belkada (Ecole Normale Superieure)
Artem Chumachenko (MynaLabs)
Pavel Samygin (Moscow Institute of Physics and Technology)
Colin Raffel (UNC Chapel Hill and Hugging Face)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors