Learning misspecified, near-deterministic surrogate models: PAC-Bayes bounds in the low data regime

Bayesian inference is provably overconfident for misspecified regression problems, as the expected loss is only an upper bound to the generalization error. However, robust misspecification-aware learning schemes are not generally available, especially in the near-deterministic limit of broad relevance to scientific simulation. We provide a solution, deriving PAC-Bayes bounds for the generalization error of misspecified surrogate models, achieved by restricting posterior distributions to a parametric family and lifting PAC analysis to hyperparameter distributions. Our bound motivates a robust learning scheme which can be efficiently implemented for high-dimensional linear models, incurring a cost comparable to ridge regression. Numerical experiments with atomic force fields and PDE integration show the resulting posteriors rectifies overconfident Bayesian parameter estimation under misspecification while remaining conservative and stable for finite sample count.