Timezone: »
Flexible Learning of Sparse Neural Networks via Constrained $L_0$ Regularization
Jose Gallego-Posada · Juan Ramirez · Akram Erraqabi
Event URL: https://openreview.net/forum?id=SUQsWPcK39P »
We propose to approach the problem of learning $L_0$-sparse networks using a constrained formulation of the optimization problem. This is in contrast to commonly used penalized approaches, which combine the regularization terms additively with the (surrogate) empirical risk. Our experiments demonstrate that we can obtain approximate solutions to the constrained optimization problem with comparable performance to state-of-the art methods for $L_0$-sparse training. Finally, we discuss how this constrained approach provides greater (hyper-)parameter interpretability and accountability from a practitioner's point of view.
We propose to approach the problem of learning $L_0$-sparse networks using a constrained formulation of the optimization problem. This is in contrast to commonly used penalized approaches, which combine the regularization terms additively with the (surrogate) empirical risk. Our experiments demonstrate that we can obtain approximate solutions to the constrained optimization problem with comparable performance to state-of-the art methods for $L_0$-sparse training. Finally, we discuss how this constrained approach provides greater (hyper-)parameter interpretability and accountability from a practitioner's point of view.
Author Information
Jose Gallego-Posada (Mila, Université de Montréal)
Juan Ramirez (Universidad EAFIT)
Akram Erraqabi (None)
More from the Same Authors
-
2022 Poster: Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints »
Jose Gallego-Posada · Juan Ramirez · Akram Erraqabi · Yoshua Bengio · Simon Lacoste-Julien -
2021 : Q & A Spotlight Presentations »
Hector Mejia · Juan Banda · Javier Orduz · Joel Cabrera Rios · Reihaneh Rabbany · Jose Gallego-Posada · Edson F. Luque -
2021 : Flexible Learning of Sparse Neural Networks via Constrained $L_0$ Regularization »
Jose Gallego-Posada · Juan Ramirez · Akram Erraqabi -
2021 Affinity Workshop: LatinX in AI (LXAI) Research @ NeurIPS 2021 »
Maria Luisa Santiago · Andres Munoz Medina · Laura Montoya · Karla Caballero Barajas · Isabel Metzger · Jose Gallego-Posada · Juan Banda · Gabriela Vega · Amanda Duarte · Patrick Feeney · Lourdes Ramírez Cerna · Walter M Mayor · Omar U. Florez · Rosina Weber · Rocio Zorrilla -
2019 : Poster Session »
Gergely Flamich · Shashanka Ubaru · Charles Zheng · Josip Djolonga · Kristoffer Wickstrøm · Diego Granziol · Konstantinos Pitas · Jun Li · Robert Williamson · Sangwoong Yoon · Kwot Sin Lee · Julian Zilly · Linda Petrini · Ian Fischer · Zhe Dong · Alexander Alemi · Bao-Ngoc Nguyen · Rob Brekelmans · Tailin Wu · Aditya Mahajan · Alexander Li · Kirankumar Shiragur · Yair Carmon · Linara Adilova · SHIYU LIU · Bang An · Sanjeeb Dash · Oktay Gunluk · Arya Mazumdar · Mehul Motani · Julia Rosenzweig · Michael Kamp · Marton Havasi · Leighton P Barnes · Zhengqing Zhou · Yi Hao · Dylan Foster · Yuval Benjamini · Nati Srebro · Michael Tschannen · Paul Rubenstein · Sylvain Gelly · John Duchi · Aaron Sidford · Robin Ru · Stefan Zohren · Murtaza Dalal · Michael A Osborne · Stephen J Roberts · Moses Charikar · Jayakumar Subramanian · Xiaodi Fan · Max Schwarzer · Nicholas Roberts · Simon Lacoste-Julien · Vinay Prabhu · Aram Galstyan · Greg Ver Steeg · Lalitha Sankar · Yung-Kyun Noh · Gautam Dasarathy · Frank Park · Ngai-Man (Man) Cheung · Ngoc-Trung Tran · Linxiao Yang · Ben Poole · Andrea Censi · Tristan Sylvain · R Devon Hjelm · Bangjie Liu · Jose Gallego-Posada · Tyler Sypherd · Kai Yang · Jan Nikolas Morshuis