Scale-Invariant Empirical-Bayes Laplace Approximation for ReLU Networks

A post-hoc Bayesian procedure should not change its evidence or uncertainty estimates under a function-preserving reparameterization. Standard empirical-Bayes (EB) Laplace approximation violates this principle for ReLU networks. Positive homogeneity induces continuous scale orbits of equivalent parameter vectors, yet the standard isotropic objective depends on the chosen representative through the prior norm and curvature log-determinant term. Equivalent ReLU parameterizations can thus yield different marginal likelihoods, selected prior precisions, posterior covariances, and linearized predictive uncertainties. We fix this by introducing $\gamma$-canonicalization, selecting a canonical representative of each ReLU scale orbit and fitting Laplace there. This is equivalent to transporting the isotropic Gaussian prior from the canonical representative back to the original coordinates, producing a precision field that transforms compatibly with the curvature. For every $\gamma\in[0,1]$, the generalized evidence and linearized predictive distribution are invariant under ReLU rescaling. The parameter $\gamma$ indexes a family of orbit-consistent priors and is selected jointly with the prior precision by marginal likelihood, making EB Laplace a well-defined procedure on ReLU scale-equivalence classes. Empirically, we show that standard EB-Laplace varies substantially across functionally identical ReLU rescalings, while $\gamma$-canonicalized Laplace collapses this variation to zero.