Spotlight
Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation
Liwei Wang · Lunjia Hu · Jiayuan Gu · Zhiqiang Hu · Yue Wu · Kun He · John Hopcroft

Thu Dec 6th 04:50 -- 04:55 PM @ Room 220 E
in Thu Track 2 -- Session 2 »

It is widely believed that learning good representations is one of the main reasons for the success of deep neural networks. Although highly intuitive, there is a lack of theory and systematic approach quantitatively characterizing what representations do deep neural networks learn. In this work, we move a tiny step towards a theory and better understanding of the representations. Specifically, we study a simpler problem: How similar are the representations learned by two networks with identical architecture but trained from different initializations. We develop a rigorous theory based on the neuron activation subspace match model. The theory gives a complete characterization of the structure of neuron activation subspace matches, where the core concepts are maximum match and simple match which describe the overall and the finest similarity between sets of neurons in two networks respectively. We also propose efficient algorithms to find the maximum match and simple matches. Finally, we conduct extensive experiments using our algorithms. Experimental results suggest that, surprisingly, representations learned by the same convolutional layers of networks trained from different initializations are not as similar as prevalently expected, at least in terms of subspace match.

Author Information

Liwei Wang (Peking University)
Lunjia Hu (Stanford University)
Jiayuan Gu (University of California, San Diego)
Zhiqiang Hu (Peking University)
Yue Wu (Peking University)
Kun He (Hua Zhong University of Science and Technology)
John Hopcroft (Cornell University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors