Competitions
AIMO Interpretability Challenge
Michal Štefánik ⋅ Philipp Mondorf ⋅ Andreas Waldis ⋅ Qianying Liu ⋅ Marek Kadlčík ⋅ Michal Spiegel ⋅ Adam Vawda-Oomerjee ⋅ Chaoran Liu ⋅ Simon Frieder ⋅ Fazl Barez ⋅ Pontus Lars Erik Saito Stenetorp ⋅ Michal Stefanik
We propose the AIMO Interpretability Challenge, a competition on distinguishing robust from spurious reasoning in frontier mathematical language models based on the models' internal behaviour. The challenge is motivated by a central limitation of standard reasoning benchmarks: strong final-answer accuracy does not reveal whether a model relies on stable reasoning mechanisms or exploits brittle reasoning shortcuts. Building on AI Mathematical Olympiad (AIMO) problems and submissions, together with resources from the Fields Model Initiative, the competition will provide (1) newly-published olympiad-level math reasoning problems and their symbolic representations, allowing generation of their novel functional variants, (2) access to frontier reasoning models, and (3) assessments of models' adversarial robustness on these problems. Participants will use these resources, along with up to 128 H200 GPUs per team that we provide, to develop methods for identifying which models solve problems robustly. Our competition will also create a new, open robustness benchmark and baseline systems, aiming to provide a lasting foundation for standard benchmarking in mathematical reasoning and interpretability. Scientifically, the competition connects interpretability and generalization research around a central question in AI research: can we determine if, and to what extent, the decision-making of frontier AI models is generalizable and thus, reliable?
Show more
FAIR Universe Weak Lensing ML Uncertainty Challenge on Distribution Shifts in Cosmology
Biwei Dai ⋅ Po-Wen Chang ⋅ Wahid Bhimji ⋅ Paolo Calafiura ⋅ Ragansu Chakkappai ⋅ Yuan-Tang Chou ⋅ Ibrahim Elsharkawy ⋅ Steven Farrell ⋅ Isabelle Guyon ⋅ Chris Harris ⋅ Benjamin Nachman ⋅ David Rousseau ⋅ Uros Seljak ⋅ Ihsan Ullah ⋅ Yulei Zhang ⋅ Wahid Bhimji
Weak gravitational lensing is a strong cosmological probe that enables direct mapping of the matter distribution and precise constraints on the fundamental parameters governing structure formation in the universe. In recent years, machine learning (ML) has shown strong potential for extracting cosmological information from weak-lensing data beyond traditional summary statistics. However, significant progress has been limited by the lack of standardized benchmarks that capture the complexity of the dataset, simulator imperfections, and various systematic biases encountered in realistic analyses. To address this gap, we introduce the FAIR Universe Weak Lensing ML Uncertainty Challenge, a series of ML competitions aimed at inferring fundamental cosmological properties from weak lensing data under potential distribution shifts, while establishing a standardized benchmark for rigorous comparison across methods. The first phase focused on cosmological parameter inference in the known unknown regime, where the training and test data share the same systematics. In realistic weak-lensing analyses, however, systematics are often unknown and may differ between training and observational data, leading to distribution shifts and biased inference. To further confront these challenges, we propose a new competition targeting out-of-distribution detection in the unknown unknown regime, where models must identify the data shifted by previously unseen systematics without prior knowledge. We provide a dedicated platform and dataset for this task, along with baseline methods including traditional power spectrum analysis, a convolutional neural network, and an autoencoder. The challenge will bring together the physics and ML communities to advance the methodologies for handling systematic uncertainties, data efficiency, and distribution shifts in weak lensing analysis with ML, ultimately facilitating the deployment of ML approaches into upcoming weak lensing surveys.
Show more
The Smart Buildings Challenge: Learning Foundation Models for Real-World Optimization at Scale
Judah Goldfeder ⋅ Philippe Wyder ⋅ Na Li ⋅ Victoria Dean ⋅ Donna Vakalis ⋅ Jiong Lin ⋅ Bing Dong ⋅ Michael J Rossetti ⋅ Nathan Kutz ⋅ Sohei Arisaka ⋅ John Sipple ⋅ Hod Lipson ⋅ Yoshua Bengio ⋅ Pmw
Buildings cause nearly 40% of carbon emissions in the US. Of total building emissions, HVAC systems are responsible for approximately 40-60% and account for 15% of global energy consumption. Despite advances in artificial intelligence, most buildings still operate as inefficiently as they did decades ago. Building optimization, perhaps more so than any other optimization problem, has enormous potential for Machine Learning (ML) driven impact on climate change, in both the short and long term. One of the biggest obstacles for building optimization is the lack of policy transfer. Most building control systems, whether simulation or learning based, don't generalize: they only work well for the building they were developed for, but not for other buildings. To solve this problem, we propose a challenge for the development of building foundation models that easily adapt to new buildings with minimal configuration and thereby enable building optimization to scale across the globe. Participants will build control policies on a set of simulated buildings, and have their policies evaluated on an unseen set of buildings. The buildings vary in size, layout, weather, material properties, and HVAC system. Successful outcomes from this initiative promise a massive leap forward in global sustainability efforts and demonstrate the great potential of ML to combat climate change.
Show more
The Steerability Challenge
Erik Miehling ⋅ Ching-Yun Ko ⋅ Matthew Riemer ⋅ Maria Chang ⋅ Pierre Dognin ⋅ Inge Vejsbjerg ⋅ Moninder Singh ⋅ Dennis Wei ⋅ Karthikeyan Natesan Ramamurthy ⋅ Pin-Yu Chen ⋅ Bharadwaj Ravichandran ⋅ Arslan Basharat ⋅ David Chan ⋅ Chen Xiong ⋅ Zhiyuan HE ⋅ Tsung-Yi Ho ⋅ Tommaso Tosato ⋅ Michael J Clark
The Steerability Challenge is a competition that tasks participants with designing steering interventions for large language models (LLMs) that achieve a target behavior while minimizing unintended side-effects. Submissions for the competition will be implemented using the AI Steerability 360 toolkit, which provides tooling for designing steering methods across various model control surfaces (prompting, activation steering, fine-tuning, and decoding) and the ability to construct compositions of interventions. The specific challenge task will require participants to design interventions that maximally reduce sycophantic behavior while minimizing the negative impact on general capabilities, as measured by a range of capability benchmarks. The competition aims to improve the community's understanding of how steering (especially under composed interventions) impacts general model behavior, and will consequently enable valuable insights into how LLMs can be reliably controlled in practice.
Show more
Successful Page Load