How Universal Are Concept Directions? A Controlled Stress-Test of Representation Engineering Under RLHF

Representation engineering extracts a single linear direction for a concept such as refusal or honesty and treats it as valid wherever the concept occurs. We stress-test this universality assumption under controlled conditions: for each concept we compare directions extracted within four semantic domains against a direction pooled across domains, on both the aligned and pre-alignment checkpoints of Qwen2.5-3B, with confounds from mixed data sources, templated responses, and domain imbalance explicitly removed. Universality holds for refusal: per-domain and global directions agree closely (cosine ≥ 0.83 at every layer) under two independent operationalizations; a global steering direction shifts held-out behavior as effectively as domain-specific ones; and probes transfer across domains almost perfectly. It fails partially for honesty, where early-layer directions diverge sharply (mean cosine 0.52; the math domain is near-orthogonal to all others), reconverge only partially with depth, and resist cross-domain probing, which marks the fragmentation as functional rather than a surface artifact. Contrary to our pre-registered hypothesis, alignment does not amplify this fragmentation; if anything it slightly consolidates directions. Finally, strong refusal fragmentation that we observed in a smaller pilot model did not replicate on the primary model under identical data, cautioning against generalizing single-model geometry findings. Universality is thus a property of particular concepts at particular depths in particular models, not of concept directions as such.