Well-calibrated building simulation models are key to reducing greenhouse gas emissions and optimizing building performance. Current calibration algorithms do not leverage data collected during previous calibration tasks. In this paper, we employ attentive neural processes (ANP) to meta-learn a distribution using multi-source data acquired during previously seen calibration tasks. The ANP informs a meta-learned Bayesian optimizer to accelerate calibration of new, unseen tasks. The few-shot nature of our proposed algorithm is demonstrated on a library of residential buildings validated by the United States Department of Energy (USDoE).