NeurIPS 2022

Timezone: »

Poster

Benchopt: Reproducible, efficient and collaborative optimization benchmarks

Thomas Moreau · Mathurin Massias · Alexandre Gramfort · Pierre Ablin · Pierre-Antoine Bannier · Benjamin Charlier · Mathieu Dagréou · Tom Dupre la Tour · Ghislain DURIF · Cassio F. Dantas · Quentin Klopfenstein · Johan Larsson · En Lai · Tanguy Lefort · Benoît Malézieux · Badr MOUFAD · Binh T. Nguyen · Alain Rakotomamonjy · Zaccharie Ramzi · Joseph Salmon · Samuel Vaiter

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #915

in Poster Session 6 »

Numerical validation is at the core of machine learning research as it allows us to assess the actual impact of new methods, and to confirm the agreement between theory and practice. Yet, the rapid development of the field poses several challenges: researchers are confronted with a profusion of methods to compare, limited transparency and consensus on best practices, as well as tedious re-implementation work. As a result, validation is often very partial, which can lead to wrong conclusions that slow down the progress of research. We propose Benchopt, a collaborative framework to automatize, publish and reproduce optimization benchmarks in machine learning across programming languages and hardware architectures. Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments. To demonstrate its broad usability, we showcase benchmarks on three standard ML tasks: $\ell_2$-regularized logistic regression, Lasso and ResNet18 training for image classification. These benchmarks highlight key practical findings that give a more nuanced view of state-of-the-art for these problems, showing that for practical evaluation, the devil is in the details.

Keywords: Optimization · benchmark · lasso · reproducibility · resnet · open source software · logistic regression

[ Paper] [ OpenReview]

Author Information

Thomas Moreau (Inria)

Mathurin Massias (Universita di Genova)

Alexandre Gramfort (Meta)

Pierre Ablin (Apple)

Pierre-Antoine Bannier (INRIA)

Benjamin Charlier (University of Montpellier)

Mathieu Dagréou (Inria Saclay)

Tom Dupre la Tour (UC Berkeley)

Ghislain DURIF (CNRS)

Cassio F. Dantas (INRAE, TETIS, Montpellier)

**Cassio Fraga Dantas** is a research scientist at [INRAE](https://www.inrae.fr/), UMR TETIS, Montpellier, France. Previously, he was a postdoctoral researcher at [IMAG](https://imag.edu.umontpellier.fr/) mathematics laboratory of Montpellier, in the probability and statistics team (EPS) and, prior to that, he was a 2-year postdoctoral researcher at [IRIT](https://www.irit.fr/) computer science laboratory of Toulouse, within the ERC project [FACTORY](http://projectfactory.irit.fr/) in 2020 and 2021. He performed his Ph.D studies at [Inria Rennes](https://www.inria.fr/fr/centre-inria-rennes-bretagne-atlantique), in the PANAMA group, and received his degree on signal, image and vision in 2019 from University of Rennes 1. Before that, in 2014, he obtained an engineering degree from the [Ecole Polytechnique](https://www.polytechnique.edu/) of Paris with a double degree and M.Sc in Electrical Engineering from [University of Campinas](https://www.fee.unicamp.br/?language=en), Brazil. He also has over two years of R\&D experience as an engineer at Idea Electronic Systems, LIP6 laboratory and Schneider Electric prior to his Ph.D studies. His recent research activities lie on the frontier between signal processing, machine learning and convex optimization, including contributions on: sparse inverse problem for image processing; matrix and tensor decomposition; and multi-dimensional data modeling.

Quentin Klopfenstein (University of Luxemburg)

Johan Larsson (Lund University)

En Lai (École Polytechnique)

Tanguy Lefort (University of Montpellier France)

- :seedling: Currently in a PhD thesis in Statistics at the University of Montpellier under the supervision of [Joseph Salmon](http://josephsalmon.eu/), [Benjamin Charlier](https://imag.umontpellier.fr/~charlier/index.php?page=index) and [Alexis Joly](http://www-sop.inria.fr/members/Alexis.Joly/wiki/pmwiki.php) (Inria) - :telescope: I am working on image classification and crowdsourced data, take into account label uncertainty and tasks difficulty (more soon) - :superhero: Also a comic books fan

Benoît Malézieux (INRIA)

Badr MOUFAD (INRIA)

Binh T. Nguyen (Telecom Paris)

Alain Rakotomamonjy (Université de Rouen Normandie Criteo AI Lab)

Zaccharie Ramzi (CNRS - ENS Ulm - Paris)

Zaccharie Ramzi is a PostDoc working on optimization in deep learning with Gabriel Peyre at ENS Ulm - CNRS in Paris. He was a PhD student working on the application of Deep Learning to MRI reconstruction under the supervision of Philippe Ciuciu and Jean-Luc Starck at NeuroSpin (CEA) in the Metric team. He is also a member of the Parietal team from Inria Saclay and the Cosmostat team from the Astrophysics Department of the CEA. Prior to this PhD, he graduated from Telecom ParisTech and ENS Cachan (M.Sc. Mathematics, Vision and Learning), and worked for 1 year as a Data Scientist at xbird, a Berlin-based startup.