Limits of SIGReg for Label-Free Model Selection

Can a distribution-matching training penalty also rank frozen representations without labels? We study SIGReg, the isotropic Gaussian regularizer in LeJEPA, as a standalone selection score. For centered Gaussian embeddings, its population value averaged over projection directions depends only on the covariance eigenvalues. Controlled mixtures then expose a limitation beyond covariance: at identity covariance, minimizing SIGReg can select a representation with chance-level accuracy over one with perfect measured linear-probe accuracy, yielding selection regret $0.892\pm0.012$. In the same mixture family, the score difference from Gaussian noise decreases sharply with dimension at the tested sample and projection budgets. Finally, rescaling unchanged features moves the raw score by $162\times$, motivating an explicit normalization convention. These results identify failure cases for standalone checkpoint selection; they do not evaluate LeJEPA's combined training loss. We distinguish population identities from finite-sample observations and outline practical checks for using SIGReg on frozen features.