What Must Pre-Training Provide? Limits of Post-Training for Installing New Architectural Mechanisms
Maria Masood ⋅ Mohammad Babakmehr
Abstract
We study the pre-training/post-training boundary using a controlled architectural instrument: a spectral modification of rotary position embeddings (RoPE) with learnable per-head amplitudes, exact identity initialization, and an exact trained-weight ablation. We prove, and verify on trained checkpoints, that its scientific status depends on which parameters remain trainable. In architectures without query--key normalization, a learnable key projection absorbs the modification exactly, so it adds no function-class expressivity during from-scratch training. With the backbone frozen, the same component becomes a genuine extension; for one RMS-matched synthetic $\gamma$, the tested diagonal rescaling and rank-$16$/rank-$64$ LoRA cannot directly write its exact folded update, although functional substitution by another route remains possible. The absorbable regime provides a theorem-backed known null for calibrating empirical evidence. Despite zero added expressivity, the learned parameters reach $\rmsg{=}0.095{\pm}0.006$, exact ablation worsens perplexity by $2.7$--$3.0\%$ on average, and interpretable spectral structure emerges reproducibly across seeds. Yet matched comparisons resolve no from-scratch performance difference, and a matched post-training control resolves no benefit. Thus activation, ablation dependence, and mechanistic structure --- even when mutually reinforcing --- do not establish added capacity or benefit. We turn this case into an actionable pre-implementation screen: before training an architectural addition, test whether parameters already being optimized can express it. A prospective five-seed application is statistically unresolved at long context, reinforcing rather than closing the need for this distinction. The broader lesson is that the parameterization available at each training stage determines whether the same code is redundant or genuinely new.
Chat is not available.
Successful Page Load