FAIR Universe Weak Lensing ML Uncertainty Challenge on Distribution Shifts in Cosmology

Weak gravitational lensing is a strong cosmological probe that enables direct mapping of the matter distribution and precise constraints on the fundamental parameters governing structure formation in the universe. In recent years, machine learning (ML) has shown strong potential for extracting cosmological information from weak-lensing data beyond traditional summary statistics. However, significant progress has been limited by the lack of standardized benchmarks that capture the complexity of the dataset, simulator imperfections, and various systematic biases encountered in realistic analyses. To address this gap, we introduce the FAIR Universe Weak Lensing ML Uncertainty Challenge, a series of ML competitions aimed at inferring fundamental cosmological properties from weak lensing data under potential distribution shifts, while establishing a standardized benchmark for rigorous comparison across methods. The first phase focused on cosmological parameter inference in the known unknown regime, where the training and test data share the same systematics. In realistic weak-lensing analyses, however, systematics are often unknown and may differ between training and observational data, leading to distribution shifts and biased inference. To further confront these challenges, we propose a new competition targeting out-of-distribution detection in the unknown unknown regime, where models must identify the data shifted by previously unseen systematics without prior knowledge. We provide a dedicated platform and dataset for this task, along with baseline methods including traditional power spectrum analysis, a convolutional neural network, and an autoencoder. The challenge will bring together the physics and ML communities to advance the methodologies for handling systematic uncertainties, data efficiency, and distribution shifts in weak lensing analysis with ML, ultimately facilitating the deployment of ML approaches into upcoming weak lensing surveys.