Weight-Space Autoencoders Transfer Knowledge Across Architectures

Model merging has established itself as a straightforward and effective method for supplying specialized models with out-of-distribution knowledge, without the need for re-training. To merge models, starting from a general, pre-trained model is often a requirement that allows methods like weight averaging or task vector arithmetic to perform well. These methods inherently operate in weight space and often pose strong assumptions on the weights' structure. In this work, we propose a self-supervised meta-learning recipe that learns a latent space via auto-encoding task vectors. To build our AEs, we adapt SANE and NiNo-- two recently successful weight-space learning inductive biases--and we meta-train them on a zoo of 8 CLIP-ViT-B/16 models. We show that these AEs can be used to transfer information across specialized minima. Furthermore, due to the architecture-agnostic nature of our proposal, our recipe can generalize to unseen models and tasks. This constitutes a zero-cost knowledge transfer operator, while retaining some model merging capabilities in its learned latent space.