Poster
Cardinality Restricted Boltzmann Machines
Kevin Swersky · Daniel Tarlow · Ilya Sutskever · Richard Zemel · Russ Salakhutdinov · Ryan Adams

Mon Dec 3rd 07:00 PM -- 12:00 AM @ Harrah’s Special Events Center 2nd Floor #None
in Poster Session and Reception »

The Restricted Boltzmann Machine (RBM) is a popular density model that is also good for extracting features. A main source of tractability in RBM models is the model's assumption that given an input, hidden units activate independently from one another. Sparsity and competition in the hidden representation is believed to be beneficial, and while an RBM with competition among its hidden units would acquire some of the attractive properties of sparse coding, such constraints are not added due to the widespread belief that the resulting model would become intractable. In this work, we show how a dynamic programming algorithm developed in 1981 can be used to implement exact sparsity in the RBM's hidden units. We then expand on this and show how to pass derivatives through a layer of exact sparsity, which makes it possible to fine-tune a deep belief network (DBN) consisting of RBMs with sparse hidden layers. We show that sparsity in the RBM's hidden layer improves the performance of both the pre-trained representations and of the fine-tuned model.

Author Information

Kevin Swersky (Google)
Daniel Tarlow (Google Brain)
Ilya Sutskever (OpenAI)
Richard Zemel (Vector Institute/University of Toronto)
Russ Salakhutdinov (Carnegie Mellon University)
Ryan Adams (Princeton University)

More from the Same Authors