Estimating Active Dimension of Training Dynamics from One Scalar Log, with an Application to Grokking

Over a stretch of training in which the update rule is fixed and the parameters keep returning near states already visited, the states they revisit form a set of dimension far below the number of parameters: the active dimension of that regime. Measuring it needs the stored trajectory, whereas a run already writes scalar logs: how much of it does one such scalar carry? We reconstruct a state space by delay coordinates and estimate its intrinsic dimension by maximum likelihood on nearest-neighbour distances. The procedure returns a plausible number on any record, including one in which nothing recurs, so it estimates the active dimension only under conditions we establish separately. On six constructed systems of known active dimension it is accurate to about one component, up to a ceiling near eight. Statistics computed from the log identify the records on which the value is not a dimension at all. They reject both standard grokking settings, so we store a compressed record of the trajectory instead. Its effective rank collapses at generalisation in the four runs trained with weight decay, then re-expands, the simplification that mechanistic accounts of grokking predict. A run that merely stops moving can collapse as deeply. Only the timing of the collapse and its reversal mark the transition. Re-run over a window matched to the transition, the estimate on an ordinary log falls at the same step in those four runs and in neither of the other two, more steeply than in randomised surrogates of each run.