Timezone: »

 
Poster
Merging Models with Fisher-Weighted Averaging
Michael S Matena · Colin Raffel

Tue Nov 29 02:00 PM -- 04:00 PM (PST) @ Hall J #116
in Poster Session 2 »

Averaging the parameters of models that have the same architecture and initialization can provide a means of combining their respective capabilities. In this paper, we take the perspective that this "merging" operation can be seen as choosing parameters that approximately maximize the joint likelihood of the posteriors of the models' parameters. Computing a simple average of the models' parameters therefore corresponds to making an isotropic Gaussian approximation to their posteriors. We develop an alternative merging procedure based on the Laplace approximation where we approximate each model's posterior as a Gaussian distribution whose precision matrix corresponds to its Fisher information. We first show that our "Fisher merging" technique provides a performance boost in settings where simple parameter averaging is currently used -- specifically, robust fine-tuning and model ensembling. Then, we compare merging to standard gradient-based transfer learning and demonstrate that merging enables a fundamentally different method for transferring capabilities across models. Specifically, we show that Fisher merging is competitive with gradient-based transfer learning approaches (while being significantly cheaper) in intermediate-task training and domain-adaptive pre-training. We also show that our merging procedure makes it possible to combine models in previously unexplored ways. We release our code to facilitate future research into methods for merging models.

Keywords: transfer learning · Ensembling · parameter averaging

Author Information

Michael S Matena (University of North Carolina at Chapel Hill)
Colin Raffel (UNC Chapel Hill and Hugging Face)

More from the Same Authors