Null Controls Are Necessary but Not Sufficient: Auditing Acceptance Guards for Scientific Discovery and Interpretability
Abstract
An automated-discovery system needs something that proposes candidates and something that decides which to keep. This paper audits the deciders, reporting three quantities for every acceptance guard: how often it accepts candidates built to carry nothing (NAR), how often it accepts known positives (PAR), and how often what it accepts survives an independent consequence test (CAR). The descriptive guards in common use accept a great deal that carries nothing. A coarse-graining that maps every state to zero closes perfectly on 96% of 129 chaotic systems from a public benchmark, while a refutable second guard rejects it on all 129. For representational correspondence the cause is structural: linear CKA gives the same score to a genuine correspondence and to that map composed with a random rotation, so more data does not help (NAR = 0.83 in-sample, 0.67 held out, 0.80 on ESM-2 protein models). A localised difference-patching consequence test refuses every null while accepting known positives (PAR = 1.00). Rejecting nulls is not sufficient: on sparse-autoencoder features the correlational guard has NAR = 0.00 yet disagrees with an independent intervention on more than half of what it accepts (CAR@150 = 0.45 against 0.005 for matched controls).