Symmetry Alignment Across Depth in Transformer Merging

Symmetry-based alignment can yield zero-barrier paths between independently trained Transformers at fixed depth. We test whether this picture extends across depth by padding the shallower model with exact identity blocks. The padded model shares the deep model’s L-block parameter space but lies in an invariant subset where selected residual branches vanish. Across five depth pairs on WikiText-2, Penn Treebank, and Tiny Shakespeare, inherited permutation-and-orthogonal (P+O) alignment removes 90.1–94.6% of the naive barrier yet leaves a positive residual of 0.098–0.244 nats in every tested configuration. Additional width (d ∈ {256, 512, 768} where available) and layer-placement ablations show that the residual depends strongly on representation width and on where identity blocks are inserted. Raw identity-slot count does not order the residuals, and matched-ratio depth trends are corpus-dependent rather than universal. Keywords: linear mode connectivity, parameter space symmetry, quotient geometry, model merging, Transformers, orthogonal group, loss landscape geometry