Layer-Wise Density Ratios Are Not Invariant Under ReLU Rescaling: A Test, a Repair, and a Critical-Dimension Statistic for Neural Network Training Stability

Dinh et al.~[1] showed that many sharpness measures fail a basic invariance test: under ReLU rescaling, the network function remains unchanged while the metric can vary. We apply the same test to layer-wise density-ratio statistics, where it has not, so far as we can establish, previously been used. Auditing nine post-hoc out-of-distribution detectors across an MLP, a CNN, and a ResNet-18, we find that most satisfy the invariance requirement exactly. Every failure arises from a dimensionful absolute constant rather than the underlying statistic itself. Fixed ReAct thresholds shift AUROC by up to 0.573, absolute Mahalanobis regularisation by 0.061, and layer-wise likelihood-ratio methods by up to 0.330. Notably, the same absolute constant can fail on some architectures yet pass on others, revealing a conditional failure mode that standard benchmark evaluations may miss. Our own initial density-ratio construction exhibits the same problem, producing invariance artefacts larger than the training-failure signal it is intended to detect. We therefore develop a repaired design that satisfies the invariance test exactly. The framework extends naturally to Transformers under their corresponding symmetry group, where we identify an additional requirement: the smoothing noise must also be equivariant. Beyond invariance, the repaired statistic provides an early-warning signal for training instability. On two of the three architectures studied, it detects gradual representational collapse before validation loss deteriorates, with average lead times of 128 training steps on the ReLU MLP and 162 steps on the MNIST CNN. Lead is measured from each metric crossing its own alarm band, calibrated on clean runs to a matched false-positive rate before any comparison. Code, run records, and negative results are released.