Workshop: The pre-registration experiment: an alternative publication model for machine learning research

Luca Bertinetto, João F. Henriques, Samuel Albanie, Michela Paganini, Gul Varol

2020-12-11T06:15:00-08:00 - 2020-12-11T14:30:00-08:00
Abstract: Machine learning research has benefited considerably from the adoption of standardised public benchmarks. In this workshop proposal, we do not argue against the importance of these benchmarks, but rather against the current incentive system and its heavy reliance upon performance as a proxy for scientific progress. The status quo incentivises researchers to “beat the state of the art”, potentially at the expense of deep scientific understanding and rigorous experimental design. Since typically only positive results are rewarded, the negative results inevitably encountered during research are often omitted, allowing many other groups to unknowingly and wastefully repeat the same negative findings. Pre-registration is a publishing and reviewing model that aims to address these issues by changing the incentive system. A pre-registered paper is a regular paper that is submitted for peer-review without any experimental results, describing instead an experimental protocol to be followed after the paper is accepted. This implies that it is important for the authors to make compelling arguments from theory or past published evidence. As for reviewers, they must assess these arguments together with the quality of the experimental design, rather than comparing numeric results. In this workshop, we propose to conduct a full pilot study in pre-registration for machine learning. It follows a successful small-scale trial of pre-registration in computer vision and is more broadly inspired by the success of pre-registration in the life sciences.



Chat is not available.


2020-12-11T06:15:00-08:00 - 2020-12-11T06:30:00-08:00
Opening Remarks
Luca Bertinetto
2020-12-11T06:30:00-08:00 - 2020-12-11T06:31:00-08:00
Introduction to Francis Bach
2020-12-11T06:31:00-08:00 - 2020-12-11T07:00:00-08:00
Francis Bach - Where is Machine Learning Going?
Francis Bach
2020-12-11T07:00:00-08:00 - 2020-12-11T07:01:00-08:00
Introduction to Yoshua Bengio
2020-12-11T07:01:00-08:00 - 2020-12-11T07:30:00-08:00
Yoshua Bengio - Incentives for Researchers
Yoshua Bengio
2020-12-11T07:30:00-08:00 - 2020-12-11T07:31:00-08:00
Introduction to oral session 1
2020-12-11T07:31:00-08:00 - 2020-12-11T07:36:00-08:00
Contributed talk - Contrastive Self-Supervised Learning for Skeleton Action Recognition
Shaoyi Du
2020-12-11T07:36:00-08:00 - 2020-12-11T07:41:00-08:00
Contributed talk - PCA Retargeting: Encoding Linear Shape Models as Convolutional Mesh Autoencoders
Eimear O' Sullivan
2020-12-11T07:41:00-08:00 - 2020-12-11T07:46:00-08:00
Contributed talk - Testing the Genomic Bottleneck Hypothesis in Hebbian Meta-Learning
Rasmus Berg Palm
2020-12-11T07:46:00-08:00 - 2020-12-11T07:51:00-08:00
Contributed talk - Policy Convergence Under the Influence of Antagonistic Agents in Markov Games
Chase Dowling
2020-12-11T08:00:00-08:00 - 2020-12-11T08:01:00-08:00
Introduction to poster session
2020-12-11T08:01:00-08:00 - 2020-12-11T09:00:00-08:00
Poster session (on
2020-12-11T09:00:00-08:00 - 2020-12-11T10:30:00-08:00
Break 1
2020-12-11T10:30:00-08:00 - 2020-12-11T10:31:00-08:00
Introduction to Joelle Pineau
2020-12-11T10:31:00-08:00 - 2020-12-11T11:00:00-08:00
Joelle Pineau - Can pre-registration lead to better reproducibility in ML research?
Joelle Pineau
2020-12-11T11:00:00-08:00 - 2020-12-11T11:01:00-08:00
Introduction to oral session 2
2020-12-11T11:01:00-08:00 - 2020-12-11T11:06:00-08:00
Contributed talk - Confronting Domain Shift in Trained Neural Networks
Cari Martinez
2020-12-11T11:06:00-08:00 - 2020-12-11T11:11:00-08:00
Contributed talk - Unsupervised Resource Allocation with Graph Neural Networks
Miles Cranmer
2020-12-11T11:11:00-08:00 - 2020-12-11T11:16:00-08:00
Contributed talk - FedPerf: A Practitioners' Guide to Performance of Federated Learning Algorithms
Tushar Semwal
2020-12-11T11:16:00-08:00 - 2020-12-11T11:21:00-08:00
Contributed talk - On the low-density latent regions of VAE-based language models
Ruizhe Li
2020-12-11T11:30:00-08:00 - 2020-12-11T11:31:00-08:00
Introduction to Jessica Zosa Forde
2020-12-11T11:31:00-08:00 - 2020-12-11T12:00:00-08:00
Jessica Zosa Forde - Build, Start, Run, Push: Computational Registration of ML Experiments
Jessica Forde
2020-12-11T12:00:00-08:00 - 2020-12-11T12:01:00-08:00
Introduction to break 2
2020-12-11T12:01:00-08:00 - 2020-12-11T12:30:00-08:00
Break 2
2020-12-11T12:30:00-08:00 - 2020-12-11T12:31:00-08:00
Introduction to Kirstie Whitaker
2020-12-11T12:31:00-08:00 - 2020-12-11T13:00:00-08:00
Kirstie Whitaker - The Turing Way: Transparent research through the scientific lifecycle
Kirstie Whitaker
2020-12-11T13:00:00-08:00 - 2020-12-11T13:01:00-08:00
Introduction to open discussion
2020-12-11T13:01:00-08:00 - 2020-12-11T14:30:00-08:00
Open Discussion
Closing remarks