Timezone: »
Restricted Boltzmann machines are undirected neural networks which have been shown tobe effective in many applications, including serving as initializations fortraining deep multi-layer neural networks. One of the main reasons for their success is theexistence of efficient and practical stochastic algorithms, such as contrastive divergence,for unsupervised training. We propose an alternative deterministic iterative procedure based on an improved mean field method from statistical physics known as the Thouless-Anderson-Palmer approach. We demonstrate that our algorithm provides performance equal to, and sometimes superior to, persistent contrastive divergence, while also providing a clear and easy to evaluate objective function. We believe that this strategycan be easily generalized to other models as well as to more accurate higher-order approximations, paving the way for systematic improvements in training Boltzmann machineswith hidden units.
Author Information
Marylou Gabrie (École Normale Supérieure, ICFP, Laboratoire de Physique Statistique )
Eric W Tramel (LPS, École Normale Supérieure)
Florent Krzakala (Ecole Normale Superieure Paris)
More from the Same Authors
-
2021 Spotlight: Learning Gaussian Mixtures with Generalized Linear Models: Precise Asymptotics in High-dimensions »
Bruno Loureiro · Gabriele Sicuro · Cedric Gerbelot · Alessandro Pacco · Florent Krzakala · Lenka Zdeborová -
2023 Poster: Optimal Algorithms for the Inhomogeneous Spiked Wigner Model »
Aleksandr Pak · Justin Ko · Florent Krzakala -
2023 Poster: Universality laws for Gaussian mixtures in generalized linear models »
Yatin Dandi · Ludovic Stephan · Florent Krzakala · Bruno Loureiro · Lenka Zdeborová -
2022 Poster: Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks »
Rodrigo Veiga · Ludovic Stephan · Bruno Loureiro · Florent Krzakala · Lenka Zdeborová -
2022 Poster: Subspace clustering in high-dimensions: Phase transitions & Statistical-to-Computational gap »
Luca Pesce · Bruno Loureiro · Florent Krzakala · Lenka Zdeborová -
2022 Poster: Multi-layer State Evolution Under Random Convolutional Design »
Max Daniels · Cedric Gerbelot · Florent Krzakala · Lenka Zdeborová -
2021 Poster: Learning Gaussian Mixtures with Generalized Linear Models: Precise Asymptotics in High-dimensions »
Bruno Loureiro · Gabriele Sicuro · Cedric Gerbelot · Alessandro Pacco · Florent Krzakala · Lenka Zdeborová -
2021 Poster: Learning curves of generic features maps for realistic datasets with a teacher-student model »
Bruno Loureiro · Cedric Gerbelot · Hugo Cui · Sebastian Goldt · Florent Krzakala · Marc Mezard · Lenka Zdeborová -
2021 Poster: Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime »
Hugo Cui · Bruno Loureiro · Florent Krzakala · Lenka Zdeborová -
2020 : Orals 2.2: Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignmen »
Julien Launay · Iacopo Poli · Laurent Daudet · Florent Krzakala -
2020 Poster: Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures »
Julien Launay · Iacopo Poli · François Boniface · Florent Krzakala -
2020 Poster: Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization »
Benjamin Aubin · Florent Krzakala · Yue Lu · Lenka Zdeborová -
2020 Poster: Reservoir Computing meets Recurrent Kernels and Structured Transforms »
Jonathan Dong · Ruben Ohana · Mushegh Rafayelyan · Florent Krzakala -
2020 Oral: Reservoir Computing meets Recurrent Kernels and Structured Transforms »
Jonathan Dong · Ruben Ohana · Mushegh Rafayelyan · Florent Krzakala -
2020 Poster: Phase retrieval in high dimensions: Statistical and computational phase transitions »
Antoine Maillard · Bruno Loureiro · Florent Krzakala · Lenka Zdeborová -
2020 Poster: Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification »
Francesca Mignacco · Florent Krzakala · Pierfrancesco Urbani · Lenka Zdeborová -
2020 Poster: Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval »
Stefano Sarao Mannelli · Giulio Biroli · Chiara Cammarota · Florent Krzakala · Pierfrancesco Urbani · Lenka Zdeborová -
2019 : Poster Session »
Jonathan Scarlett · Piotr Indyk · Ali Vakilian · Adrian Weller · Partha P Mitra · Benjamin Aubin · Bruno Loureiro · Florent Krzakala · Lenka Zdeborová · Kristina Monakhova · Joshua Yurtsever · Laura Waller · Hendrik Sommerhoff · Michael Moeller · Rushil Anirudh · Shuang Qiu · Xiaohan Wei · Zhuoran Yang · Jayaraman Thiagarajan · Salman Asif · Michael Gillhofer · Johannes Brandstetter · Sepp Hochreiter · Felix Petersen · Dhruv Patel · Assad Oberai · Akshay Kamath · Sushrut Karmalkar · Eric Price · Ali Ahmed · Zahra Kadkhodaie · Sreyas Mohan · Eero Simoncelli · Carlos Fernandez-Granda · Oscar Leong · Wesam Sakla · Rebecca Willett · Stephan Hoyer · Jascha Sohl-Dickstein · Sam Greydanus · Gauri Jagatap · Chinmay Hegde · Michael Kellman · Jonathan Tamir · Nouamane Laanait · Ousmane Dia · Mirco Ravanelli · Jonathan Binas · Negar Rostamzadeh · Shirin Jalali · Tiantian Fang · Alex Schwing · Sébastien Lachapelle · Philippe Brouillard · Tristan Deleu · Simon Lacoste-Julien · Stella Yu · Arya Mazumdar · Ankit Singh Rawat · Yue Zhao · Jianshu Chen · Xiaoyang Li · Hubert Ramsauer · Gabrio Rizzuti · Nikolaos Mitsakos · Dingzhou Cao · Thomas Strohmer · Yang Li · Pei Peng · Gregory Ongie -
2019 Poster: The spiked matrix model with generative priors »
Benjamin Aubin · Bruno Loureiro · Antoine Maillard · Florent Krzakala · Lenka Zdeborová -
2016 Poster: Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula »
jean barbier · Mohamad Dia · Nicolas Macris · Florent Krzakala · Thibault Lesieur · Lenka Zdeborová -
2015 Poster: Matrix Completion from Fewer Entries: Spectral Detectability and Rank Estimation »
Alaa Saade · Florent Krzakala · Lenka Zdeborová -
2014 Poster: Spectral Clustering of graphs with the Bethe Hessian »
Alaa Saade · Florent Krzakala · Lenka Zdeborová