No Location Left Behind: Introducing the Fairness Assessment for Implicit Representations of Earth Data

Encoding and predicting physical measurements such as temperature or carbon dioxide is instrumental to many high-stakes challenges -- including climate change. Yet, all recent advances solely assess models' performances at a global scale. But while models' predictions are improving on average over the entire globe, performances on sub-groups such as islands or coastal areas are left uncharted. To ensure safe deployment of those models, we thus introduce FAIR-Earth, a fine-grained evaluation suite made of diverse and high-resolution dataset. Our findings are striking--current methods produce highly biased predictions towards specific geospatial locations. The specifics of the biases vary based on the data modality and hyper-parameters of the models. Hence, we hope that FAIR-Earth will enable future research to design solutions aware of those per-group biases.