A Label-Noise-Gated Sign Flip in Validation-Based Early Stopping

An autonomous experiment loop can decide, without human review of each run, when to stop training a model using a held-out validation split. We study whether that decision is reliably beneficial. Using a fully autonomous research loop, we ran 750 controlled training runs of a two-layer MLP on sklearn's digit-recognition dataset, sweeping validation-set size $v \in \{10,20,50,100,150\}$ and symmetric label-noise rate $p \in \{0,0.1,0.2,0.3,0.4\}$ with 30 seeds per cell, and compared validation-selected stopping against a naive fixed-epoch-budget baseline that ignores the validation split entirely. Qualifying result: the sign of the gap between the two policies flips with label noise. At $p=0$, validation-based stopping loses to the fixed-budget baseline at every tested validation size (win rate 3--13\%, Holm-corrected Wilcoxon $p<0.002$ in all five cells), because the network exhibits benign overfitting and test accuracy does not meaningfully decline within the training budget, so any early exit yields no benefit and, in the large majority of seeds, a measurable loss. At $p \geq 0.2$, validation-based stopping wins with high probability (70--87\% at $v=10$, $\geq$93\% at $v \geq 50$); its margin grows with $p$ throughout, and with $v$ up to $v \approx 50$--$100$, past which it narrows again (e.g.\ $+0.185$ at $v{=}100$ vs.\ $+0.160$ at $v{=}150$ for $p=0.4$). At the smallest budget and lowest nonzero noise level tested ($v=10$, $p=0.1$), the advantage is not statistically distinguishable from zero. The sign flip replicates at two additional architectures not used to derive it. The result was reached after an initial experiment configuration produced an uninformative null regime; the agent diagnosed the cause, changed the configuration, and recovered a different and better-supported finding than the one originally hypothesized.