Poster
in
Workshop: UniReps: Unifying Representations in Neural Models
Model Merging by Gradient Matching
Nico Daheim · Thomas Möllenhoff · Edoardo Maria Ponti · Iryna Gurevych · Mohammad Emtiyaz Khan
[
Abstract
]
[ Project Page ]
presentation:
UniReps: Unifying Representations in Neural Models
Fri 15 Dec 6:15 a.m. PST — 3:15 p.m. PST
[
OpenReview]
Fri 15 Dec 6:15 a.m. PST — 3:15 p.m. PST
Abstract:
Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weighted-averaging to mismatches in the gradients and propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. The connection also reveals implicit assumptions in other schemes such as averaging, task arithmetic, and Fisher-weighted averaging.
Chat is not available.