Timezone: »
Deep neural networks progressively transform their inputs across multiple processing layers. What are the geometrical properties of the representations learned by these networks? Here we study the intrinsic dimensionality (ID) of data representations, i.e. the minimal number of parameters needed to describe a representation. We find that, in a trained network, the ID is orders of magnitude smaller than the number of units in each layer. Across layers, the ID first increases and then progressively decreases in the final layers. Remarkably, the ID of the last hidden layer predicts classification accuracy on the test set. These results can neither be found by linear dimensionality estimates (e.g., with principal component analysis), nor in representations that had been artificially linearized. They are neither found in untrained networks, nor in networks that are trained on randomized labels. This suggests that neural networks that can generalize are those that transform the data into low-dimensional, but not necessarily flat manifolds.
Author Information
Alessio Ansuini (International School for Advanced Studies (SISSA))
I am a theoretical physicist with a broad interest in biology, neuroscience and artificial intelligence. I am currently working on methods to extract high-level information from biological data, e.g. by analyzing representations in deep learning models, with the aim to make sense of these data and constrain theoretical models. I live in Trieste, Italy, and work as a researcher in Area Science Park.
Alessandro Laio (International School for Advanced Studies (SISSA))
Jakob H Macke (Technical University of Munich, Munich, Germany)
Davide Zoccolan (Visual Neuroscience Lab, International School for Advanced Studies (SISSA))
More from the Same Authors
-
2022 : The geometry of hidden representations of protein language models »
Lucrezia Valeriani · Francesca Cuturello · Alessio Ansuini · Alberto Cazzaniga -
2022 Poster: Prune and distill: similar reformatting of image information along rat visual cortex and deep neural networks »
Paolo Muratore · Sina Tafazoli · Eugenio Piasini · Alessandro Laio · Davide Zoccolan -
2022 Poster: Redundant representations help generalization in wide neural networks »
Diego Doimo · Aldo Glielmo · Sebastian Goldt · Alessandro Laio -
2022 Poster: Truncated proposals for scalable and hassle-free simulation-based inference »
Michael Deistler · Pedro Goncalves · Jakob H Macke -
2022 Poster: Efficient identification of informative features in simulation-based inference »
Jonas Beck · Michael Deistler · Yves Bernaerts · Jakob H Macke · Philipp Berens -
2020 Poster: Hierarchical nucleation in deep neural networks »
Diego Doimo · Aldo Glielmo · Alessio Ansuini · Alessandro Laio -
2019 : Poster Session »
Pravish Sainath · Mohamed Akrout · Charles Delahunt · Nathan Kutz · Guangyu Robert Yang · Joseph Marino · L F Abbott · Nicolas Vecoven · Damien Ernst · andrew warrington · Michael Kagan · Kyunghyun Cho · Kameron Harris · Leopold Grinberg · John J. Hopfield · Dmitry Krotov · Taliah Muhammad · Erick Cobos · Edgar Walker · Jacob Reimer · Andreas Tolias · Alexander Ecker · Janaki Sheth · Yu Zhang · Maciej Wołczyk · Jacek Tabor · Szymon Maszke · Roman Pogodin · Dane Corneil · Wulfram Gerstner · Baihan Lin · Guillermo Cecchi · Jenna M Reinen · Irina Rish · Guillaume Bellec · Darjan Salaj · Anand Subramoney · Wolfgang Maass · Yueqi Wang · Ari Pakman · Jin Hyung Lee · Liam Paninski · Bryan Tripp · Colin Graber · Alex Schwing · Luke Prince · Gabriel Ocker · Michael Buice · Benjamin Lansdell · Konrad Kording · Jack Lindsey · Terrence Sejnowski · Matthew Farrell · Eric Shea-Brown · Nicolas Farrugia · Victor Nepveu · Jiwoong Im · Kristin Branson · Brian Hu · Ramakrishnan Iyer · Stefan Mihalas · Sneha Aenugu · Hananel Hazan · Sihui Dai · Tan Nguyen · Doris Tsao · Richard Baraniuk · Anima Anandkumar · Hidenori Tanaka · Aran Nayebi · Stephen Baccus · Surya Ganguli · Dean Pospisil · Eilif Muller · Jeffrey S Cheng · Gaël Varoquaux · Kamalaker Dadi · Dimitrios C Gklezakos · Rajesh PN Rao · Anand Louis · Christos Papadimitriou · Santosh Vempala · Naganand Yadati · Daniel Zdeblick · Daniela M Witten · Nicholas Roberts · Vinay Prabhu · Pierre Bellec · Poornima Ramesh · Jakob H Macke · Santiago Cadena · Guillaume Bellec · Franz Scherr · Owen Marschall · Robert Kim · Hannes Rapp · Marcio Fonseca · Oliver Armitage · Jiwoong Im · Thomas Hardcastle · Abhishek Sharma · Wyeth Bair · Adrian Valente · Shane Shang · Merav Stern · Rutuja Patil · Peter Wang · Sruthi Gorantla · Peter Stratton · Tristan Edwards · Jialin Lu · Martin Ester · Yurii Vlasov · Siavash Golkar -
2017 Spotlight: Fast amortized inference of neural activity from calcium imaging data with variational autoencoders »
Artur Speiser · Jinyao Yan · Evan Archer · Lars Buesing · Srinivas C Turaga · Jakob H Macke -
2017 Poster: Fast amortized inference of neural activity from calcium imaging data with variational autoencoders »
Artur Speiser · Jinyao Yan · Evan Archer · Lars Buesing · Srinivas C Turaga · Jakob H Macke -
2017 Poster: Extracting low-dimensional dynamics from multiple large-scale neural population recordings by learning to predict correlations »
Marcel Nonnenmacher · Srinivas C Turaga · Jakob H Macke -
2017 Poster: Flexible statistical inference for mechanistic models of neural dynamics »
Jan-Matthis Lueckmann · Pedro Goncalves · Giacomo Bassetto · Kaan Öcal · Marcel Nonnenmacher · Jakob H Macke -
2016 : From Brains to Bits and Back Again »
Yoshua Bengio · Terrence Sejnowski · Christos H Papadimitriou · Jakob H Macke · Demis Hassabis · Alyson Fletcher · Andreas Tolias · Jascha Sohl-Dickstein · Konrad P Koerding -
2016 Workshop: Brains and Bits: Neuroscience meets Machine Learning »
Alyson Fletcher · Eva Dyer · Jascha Sohl-Dickstein · Joshua T Vogelstein · Konrad Koerding · Jakob H Macke -
2015 : Correlations and Signatures of Criticality in Neural Population Models »
Jakob H Macke -
2015 Workshop: Statistical Methods for Understanding Neural Systems »
Alyson Fletcher · Jakob H Macke · Ryan Adams · Jascha Sohl-Dickstein -
2015 Poster: Unlocking neural population non-stationarities using hierarchical dynamics models »
Mijung Park · Gergo Bohner · Jakob H Macke -
2014 Workshop: Large scale optical physiology: From data-acquisition to models of neural coding »
Il Memming Park · Jakob H Macke · Ferran Diego Andilla · Eftychios Pnevmatikakis · Jeremy Freeman -
2014 Poster: A Bayesian model for identifying hierarchically organised states in neural population activity »
Patrick Putzky · Florian Franzen · Giacomo Bassetto · Jakob H Macke -
2014 Spotlight: A Bayesian model for identifying hierarchically organised states in neural population activity »
Patrick Putzky · Florian Franzen · Giacomo Bassetto · Jakob H Macke -
2014 Poster: Low-dimensional models of neural population activity in sensory cortical circuits »
Evan Archer · Urs Koster · Jonathan W Pillow · Jakob H Macke -
2013 Workshop: Acquiring and Analyzing the Activity of Large Neural Ensembles »
Srinivas C Turaga · Lars Buesing · Maneesh Sahani · Jakob H Macke -
2013 Poster: Inferring neural population dynamics from multiple partial recordings of the same neural circuit »
Srinivas C Turaga · Lars Buesing · Adam M Packer · Henry Dalgleish · Noah Pettit · Michael Hausser · Jakob H Macke -
2013 Spotlight: Inferring neural population dynamics from multiple partial recordings of the same neural circuit »
Srinivas C Turaga · Lars Buesing · Adam M Packer · Henry Dalgleish · Noah Pettit · Michael Hausser · Jakob H Macke -
2012 Poster: Spectral learning of linear dynamics from generalised-linear observations with application to neural population data »
Lars Buesing · Jakob H Macke · Maneesh Sahani -
2012 Oral: Spectral learning of linear dynamics from generalised-linear observations with application to neural population data »
Lars Buesing · Jakob H Macke · Maneesh Sahani -
2011 Oral: Empirical models of spiking in neural populations »
Jakob H Macke · Lars Buesing · John P Cunningham · Byron M Yu · Krishna V Shenoy · Maneesh Sahani -
2011 Poster: Empirical models of spiking in neural populations »
Jakob H Macke · Lars Buesing · John P Cunningham · Byron M Yu · Krishna V Shenoy · Maneesh Sahani -
2011 Poster: How biased are maximum entropy models? »
Jakob H Macke · Iain Murray · Peter E Latham -
2009 Poster: Bayesian estimation of orientation preference maps »
Jakob H Macke · Sebastian Gerwinn · Leonard White · Matthias Kaschube · Matthias Bethge -
2007 Oral: Bayesian Inference for Spiking Neuron Models with a Sparsity Prior »
Sebastian Gerwinn · Jakob H Macke · Matthias Seeger · Matthias Bethge -
2007 Poster: Bayesian Inference for Spiking Neuron Models with a Sparsity Prior »
Sebastian Gerwinn · Jakob H Macke · Matthias Seeger · Matthias Bethge -
2007 Poster: Receptive Fields without Spike-Triggering »
Jakob H Macke · Günther Zeck · Matthias Bethge -
2006 Poster: Inducing Metric Violations in Human Similarity Judgements »
Julian Laub · Jakob H Macke · Klaus-Robert Müller · Felix A Wichmann