Timezone: »

The pre-registration experiment: an alternative publication model for machine learning research
Luca Bertinetto · João Henriques · Samuel Albanie · Michela Paganini · Gul Varol

Fri Dec 11 06:15 AM -- 02:30 PM (PST) @ None
Event URL: https://preregister.science »

Machine learning research has benefited considerably from the adoption of standardised public benchmarks. In this workshop proposal, we do not argue against the importance of these benchmarks, but rather against the current incentive system and its heavy reliance upon performance as a proxy for scientific progress. The status quo incentivises researchers to “beat the state of the art”, potentially at the expense of deep scientific understanding and rigorous experimental design. Since typically only positive results are rewarded, the negative results inevitably encountered during research are often omitted, allowing many other groups to unknowingly and wastefully repeat the same negative findings. Pre-registration is a publishing and reviewing model that aims to address these issues by changing the incentive system. A pre-registered paper is a regular paper that is submitted for peer-review without any experimental results, describing instead an experimental protocol to be followed after the paper is accepted. This implies that it is important for the authors to make compelling arguments from theory or past published evidence. As for reviewers, they must assess these arguments together with the quality of the experimental design, rather than comparing numeric results. In this workshop, we propose to conduct a full pilot study in pre-registration for machine learning. It follows a successful small-scale trial of pre-registration in computer vision and is more broadly inspired by the success of pre-registration in the life sciences.

Fri 6:15 a.m. - 6:30 a.m.
Opening Remarks
Luca Bertinetto
Fri 6:31 a.m. - 7:00 a.m.
Francis Bach - Where is Machine Learning Going? (Invited talk)   
Francis Bach
Fri 7:01 a.m. - 7:30 a.m.
Yoshua Bengio - Incentives for Researchers (Invited talk)   
Yoshua Bengio
Fri 7:31 a.m. - 7:36 a.m.
Contributed talk - Contrastive Self-Supervised Learning for Skeleton Action Recognition (Contributed talk)   
Shaoyi Du
Fri 7:36 a.m. - 7:41 a.m.
Contributed talk - PCA Retargeting: Encoding Linear Shape Models as Convolutional Mesh Autoencoders (Contributed talk)   
Eimear O' Sullivan
Fri 7:41 a.m. - 7:46 a.m.
Contributed talk - Testing the Genomic Bottleneck Hypothesis in Hebbian Meta-Learning (Contributed talk)   
Rasmus Berg Palm
Fri 7:46 a.m. - 7:51 a.m.
Contributed talk - Policy Convergence Under the Influence of Antagonistic Agents in Markov Games (Contributed talk)   
Chase Dowling
Fri 8:01 a.m. - 9:00 a.m.
Poster session (on gather.town) (Poster session)  link »
Fri 9:00 a.m. - 10:30 a.m.
Break 1 (Break)
Fri 10:31 a.m. - 11:00 a.m.
Joelle Pineau - Can pre-registration lead to better reproducibility in ML research? (Invited talk)   
Joelle Pineau
Fri 11:01 a.m. - 11:06 a.m.
Contributed talk - Confronting Domain Shift in Trained Neural Networks (Contributed talk)   
Cari Martinez
Fri 11:06 a.m. - 11:11 a.m.
Contributed talk - Unsupervised Resource Allocation with Graph Neural Networks (Contributed talk)   
Miles Cranmer
Fri 11:11 a.m. - 11:16 a.m.
Contributed talk - FedPerf: A Practitioners' Guide to Performance of Federated Learning Algorithms (Contributed talk)   
Tushar Semwal
Fri 11:16 a.m. - 11:21 a.m.
Contributed talk - On the low-density latent regions of VAE-based language models (Contributed talk)   
Ruizhe Li
Fri 11:31 a.m. - 12:00 p.m.
Jessica Zosa Forde - Build, Start, Run, Push: Computational Registration of ML Experiments (Invited talk)
Jessica Forde
Fri 12:00 p.m. - 12:01 p.m.
Introduction to break 2 (Introduction)
Fri 12:01 p.m. - 12:30 p.m.
Break 2 (Break)
Fri 12:31 p.m. - 1:00 p.m.
Kirstie Whitaker - The Turing Way: Transparent research through the scientific lifecycle (Invited talk)   
Kirstie Whitaker
Fri 1:01 p.m. - 2:30 p.m.
Open Discussion (Discussion panel)
Closing remarks

Author Information

Luca Bertinetto (FiveAI Ltd.)
João Henriques (University of Oxford)
Samuel Albanie (Oxford University)
Michela Paganini (Facebook)
Gul Varol (Ecole des Ponts ParisTech)

More from the Same Authors