The Marauder’s Map: Bézier Manifolds Reveal Hidden Surfaces for Model Merging and Ensembling

Neural circuits are infamously more robust than their deep network counterparts. Biology often explains this through degeneracy, where structurally distinct configurations support the same function. Deep learning studies a parallel phenomenon through mode connectivity, which links same-objective networks by low-loss curves, sheets, or volumes. Yet real biological systems undergo continual structural change, a more flexible form of degeneracy in which specialized circuits gradually adapt toward new competencies through localized changes while preserving existing function. The analogous question in deep learning is harder than ordinary mode connectivity, asking whether many independently initialized models trained on distinct tasks can be connected by a high-dimensional region of competent multi-task solutions. We introduce Nimbus, which learns a high-dimensional Bézier manifold connecting N specialists, going beyond the one-dimensional paths and two-dimensional sheets of prior mode connectivity work. A naive parameterization would scale combinatorially with N, but Nimbus requires only a small number of learned correction tensors shared per interaction order, showing that the connecting manifold has surprisingly low complexity yet supports multi-task competencies. The resulting manifold from Nimbus forms a navigable "Marauder's Map" connecting independently trained specialists through a shared geometry of competent solutions. Across vision (CIFAR-100) and language modeling (five disjoint text domains), Nimbus produces manifolds that are both functionally and locally weight-flat. Ensembles sampled from them degrade more gracefully than baselines under input and weight perturbations, with the largest gains in negative log-likelihood and expected calibration error. Beyond its empirical behavior, Nimbus supplies a deep learning interpretation of flexible degeneracy, recasting the connectivity between specialized circuits from a low-dimensional path between distant solutions into a high-dimensional volume of nearby ones.