Ensemble Mashups: A Simple Recipe For Better Bayesian Optimization

Bayesian optimization (BO) is a popular approach to optimizing costly, black-box functions that rely on a statistical surrogate model of the function, typically a Gaussian process (GP), and the so-called acquisition function (AF). Although the choices of the GP kernel and the AF can strongly affect the results, there does not exist an automatic way of selecting them. Ensembling, namely, using several, different kernels (Multi-Model) or AFs (Multi-AF) is one possibility for deriving a BO algorithm that is robust and safer. These ideas have been considered separately in the past. In this work, we consider ensembles of both kernels and AFs (Multi-Model Multi-AF) and perform an empirical comparison to show their superiority with respect to single-ensemble algorithms.