Reconstructing High-Resolution Hyperparameter Loss Landscapes via Active Surrogate Modeling

The evaluation of deep learning models typically relies on identifying a single hyperparameter configuration that minimizes validation loss. While this approach establishes benchmark performance, it obscures the topological structure of the hyperparameter space. Theoretical insights from weight-space geometry suggest that broader minima tend to correlate with better generalization, yet extending this analysis to hyperparameter space is computationally challenging because each point on the surface requires a separate training run. In this work, we study hyperparameter selection as a landscape reconstruction problem rather than only a black-box minimization problem. We propose an active surrogate modeling framework that uses a Gaussian Process and a variance-weighted acquisition rule to map local minima, basin boundaries, and high-uncertainty regions under a limited evaluation budget. To visualize high-dimensional search spaces, we introduce an optimization-trajectory Principal Component Analysis (PCA) projection that builds a data-adaptive two-dimensional plane around the empirically useful basin. Experiments across vision and language architectures suggest that the proposed method reconstructs landscape topography with high fidelity using roughly 4–5% of the compute required for dense grid search. The resulting surfaces also reveal cases where a configuration with slightly worse in-distribution validation loss lies in a substantially broader basin and yields better out-of-distribution robustness, indicating that local topological properties can complement point-estimate validation loss in model selection.