When Attribution and Intervention Diverge in Gender-Conditioned Machine Translation
Yanan Long
Abstract
Mechanistic interpretability often proceeds in three steps: use attribution to rank candidate internal states, intervene on the strongest candidates, and infer something about the localized computation from the resulting behavioral change. We test how reliably these steps compose in gender-conditioned translation with TranslateGemma 4B. Study I compares first-order attribution with exact residual-stream donor patching at all 34 end-of-input layers for four matched English-to-German source families. Across 136 family-layer cells, attribution magnitude correlates weakly with exact intervention magnitude (Spearman $\rho=6.90\times10^{-2}$), the attribution top two never match the exact top two, and sign agreement is 45.2\% over 115 nonzero exact effects. Some exact effects also change under mathematically equivalent batching and sequence-padding choices. Study II bypasses attribution during site selection and directly attenuates residual states while scoring feminine, masculine, and inclusive English-to-French references. The predeclared matched-site comparison misses its threshold by $2.60\times10^{-3}$ nats/token. At the frozen fallback site, the protected inclusive-reference contrast has median $\Delta H=9.90\times10^{-4}$ with a 4/4 sign split at $\alpha=0.9$, and weaker attenuation does not reveal a coherent dose response. The two studies therefore separate four sources of uncertainty that are often conflated: proxy ranking, intervention robustness, perturbation specificity, and endpoint construction. For Queer NLP, the key issue is construct validity: a dataset-provided inclusive reference can serve as a downstream diagnostic without constituting a stable scalar extension of a masculine--feminine contrast.
Chat is not available.
Successful Page Load