Poster
in
Workshop: UniReps: Unifying Representations in Neural Models
Semi-Ensemble: A Simple Approach Over-parameterize Model Interpolation
Jiwoon Lee · Jaeho Lee
Fri 15 Dec 6:15 a.m. PST — 3:15 p.m. PST
We develop a unified framework for interpolating two models with various degrees of over-parameterization, having model merging and model ensemble as special cases. Instead of directly interpolating models in their original parameter space, the proposed Semi-Ensemble interpolates the over-parameterized versions of the models in a higher-dimensional joint parameter space. Here, the over-parameterizations recover each endpoint model when projected to some low-dimensional subspace spanned by a fraction of bases. By carefully constructing the joint parameter space, the interpolated model can achieve a smooth tradeoff between the total number of parameters and the model accuracy, outperforming existing baselines. Intriguingly, we show that Semi-ensembles can sometimes achieve a better performance than vanilla ensembles, even with a slightly smaller number of parameters.