Representational Drift as a Mechanism of Length-Generalization Failure
Andrew Mah ⋅ Alex Williams
Abstract
Length generalization remains a challenge for recurrent sequence models. We identify a failure mode in linear RNNs in which task-relevant representations remain linearly decodable but drift coherently through hidden-state space over long sequences. Ablating the drift-associated dimension restores long-horizon performance without retraining, providing a geometric mechanism for unexplored-state generalization failure.
Chat is not available.
Successful Page Load