Rules Amortize, Pairings Don’t: Linguistic Structure Determines What Latent Task Representations Can Replace In-Context Learning

In-context learning (ICL) can be amortized into latent objects – task vectors, func- tion vectors, context vectors – that recover few-shot behavior at zero-shot infer- ence cost, but recent theory shows a static vector acts as a single synthetic demon- stration and must fail on high-rank mappings such as word-level bijections. We ask a linguistic version of this question: which linguistic operations can be amor- tized out of the prompt? We train a 2.6M-parameter network that reads the ge- ometry of a few-shot support set (centroid, principal subspace, spectrum – com- puted once and cached) and produces an input-conditioned additive update to the query’s residual stream at a mid-depth layer of a frozen GPT-2-large/XL. Across eight inflectional directions and one lexical relation – under a canonical split that bars inverted-pair leakage between directions – three regimes emerge. On forward inflection, where 10-shot ICL is strong (0.67–0.89) and extracted task vectors col- lapse (≤0.06), the transform matches ICL at strictly zero-shot per-query cost. On lemmatization directions, which frozen GPT-2 can execute but 10 demonstrations systematically fail to convey (ICL 0.13–0.48 at 1.5B), the transform is not capped by ICL at all: it reaches 0.78–0.92, up to +72 points over ICL (past→present: 0.85 vs. 0.13). On arbitrary pairings (antonymy) every amortizer plateaus near half of ICL at every scale, capacity, and seed tested. Controls show the support mani- fold acts as a causally necessary task fingerprint – wrong-task manifolds collapse accuracy to ≤0.06, query-only variants cannot disambiguate tasks sharing an in- put space, and leave-one-task-out transfer is zero. Productive rules amortize into latent task representations – sometimes better than prompting can convey them; memorized pairings do not