Calibrating Simulators from Pairwise Preferences over Simulated Outputs

Calibrating the parameters of mechanistic simulators such that their outputs are realistic is a core problem in simulation science. Approaches for doing so typically require either or both (a) precise numerical or mathematical input from the modeller or a domain expert – such as summary statistics, a loss function, or statistics describing a belief distribution – and (b) explicit reference data. The former of these are generally difficult to specify precisely, while the latter may not exist or be readily available. To enable simulator calibration in their absence, we consider how expert judgement, supplied by human or AI, about the relative realism of simulated outputs can be used to construct parameter distributions that induce realistic simulator behaviour. In particular, we propose a scheme for constructing such distributions from binary data expressing pairwise preferences over simulated outputs. We present numerical experiments demonstrating the efficacy of the method, and discuss the limitations and potential directions for future work.