Machine learning research has benefited considerably from the adoption of standardised public benchmarks. In this workshop proposal, we do not argue against the importance of these benchmarks, but rather against the current incentive system and its heavy reliance upon performance as a proxy for scientific progress. The status quo incentivises researchers to “beat the state of the art”, potentially at the expense of deep scientific understanding and rigorous experimental design. Since typically only positive results are rewarded, the negative results inevitably encountered during research are often omitted, allowing many other groups to unknowingly and wastefully repeat the same negative findings. Pre-registration is a publishing and reviewing model that aims to address these issues by changing the incentive system. A pre-registered paper is a regular paper that is submitted for peer-review without any experimental results, describing instead an experimental protocol to be followed after the paper is accepted. This implies that it is important for the authors to make compelling arguments from theory or past published evidence. As for reviewers, they must assess these arguments together with the quality of the experimental design, rather than comparing numeric results. In this workshop, we propose to conduct a full pilot study in pre-registration for machine learning. It follows a successful small-scale trial of pre-registration in computer vision and is more broadly inspired by the success of pre-registration in the life sciences.