Timezone: »
Many recent studies have employed task-based modeling with recurrent neural networks (RNNs) to infer the computational function of different brain regions. These models are often assessed by quantitatively comparing the low-dimensional neural dynamics of the model and the brain, for example using canonical correlation analysis (CCA). However, the nature of the detailed neurobiological inferences one can draw from such efforts remains elusive. For example, to what extent does training neural networks to solve simple tasks, prevalent in neuroscientific studies, uniquely determine the low-dimensional dynamics independent of neural architectures? Or alternatively, are the learned dynamics highly sensitive to different neural architectures? Knowing the answer to these questions has strong implications on whether and how to use task-based RNN modeling to understand brain dynamics. To address these foundational questions, we study populations of thousands of networks of commonly used RNN architectures trained to solve neuroscientifically motivated tasks and characterize their low-dimensional dynamics via CCA and nonlinear dynamical systems analysis. We find the geometry of the dynamics can be highly sensitive to different network architectures, and further find striking dissociations between geometric similarity as measured by CCA and network function, yielding a cautionary tale. Moreover, we find that while the geometry of neural dynamics can vary greatly across architectures, the underlying computational scaffold: the topological structure of fixed points, transitions between them, limit cycles, and linearized dynamics, often appears {\it universal} across all architectures. Overall, this analysis of universality and individuality across large populations of RNNs provides a much needed foundation for interpreting quantitative measures of dynamical similarity between RNN and brain dynamics.
Author Information
Niru Maheswaranathan (Google Brain)
Alex Williams (Stanford University)
Matthew Golub (Stanford University)
Surya Ganguli (Stanford)
David Sussillo (Google Inc.)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: Universality and individuality in neural dynamics across large populations of recurrent networks »
Thu Dec 12th 06:45 -- 08:45 PM Room East Exhibition Hall B + C
More from the Same Authors
-
2020 Poster: Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel »
Stanislav Fort · Gintare Karolina Dziugaite · Mansheej Paul · Sepideh Kharaghani · Daniel Roy · Surya Ganguli -
2020 Poster: Point process models for sequence detection in high-dimensional neural spike trains »
Alex Williams · Anthony Degleris · Yixin Wang · Scott Linderman -
2020 Oral: Point process models for sequence detection in high-dimensional neural spike trains »
Alex Williams · Anthony Degleris · Yixin Wang · Scott Linderman -
2020 Poster: Predictive coding in balanced neural networks with noise, chaos and delays »
Jonathan Kadmon · Jonathan Timcheck · Surya Ganguli -
2020 Poster: Identifying Learning Rules From Neural Network Observables »
Aran Nayebi · Sanjana Srivastava · Surya Ganguli · Daniel Yamins -
2020 Spotlight: Identifying Learning Rules From Neural Network Observables »
Aran Nayebi · Sanjana Srivastava · Surya Ganguli · Daniel Yamins -
2020 Poster: Pruning neural networks without any data by iteratively conserving synaptic flow »
Hidenori Tanaka · Daniel Kunin · Daniel Yamins · Surya Ganguli -
2019 Poster: A unified theory for the origin of grid cells through the lens of pattern formation »
Ben Sorscher · Gabriel Mel · Surya Ganguli · Samuel Ocko -
2019 Spotlight: A unified theory for the origin of grid cells through the lens of pattern formation »
Ben Sorscher · Gabriel Mel · Surya Ganguli · Samuel Ocko -
2019 Poster: From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction »
Hidenori Tanaka · Aran Nayebi · Niru Maheswaranathan · Lane McIntosh · Stephen Baccus · Surya Ganguli -
2019 Poster: Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics »
Niru Maheswaranathan · Alex Williams · Matthew Golub · Surya Ganguli · David Sussillo -
2018 Poster: The emergence of multiple retinal cell types through efficient coding of natural movies »
Samuel Ocko · Jack Lindsey · Surya Ganguli · Stephane Deny -
2018 Poster: Statistical mechanics of low-rank tensor decomposition »
Jonathan Kadmon · Surya Ganguli -
2018 Poster: Task-Driven Convolutional Recurrent Models of the Visual System »
Aran Nayebi · Daniel Bear · Jonas Kubilius · Kohitij Kar · Surya Ganguli · David Sussillo · James J DiCarlo · Daniel Yamins -
2017 Poster: Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net »
Anirudh Goyal ALIAS PARTH GOYAL · Nan Rosemary Ke · Surya Ganguli · Yoshua Bengio -
2017 Poster: Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice »
Jeffrey Pennington · Samuel Schoenholz · Surya Ganguli -
2016 Poster: An Online Sequence-to-Sequence Model Using Partial Conditioning »
Navdeep Jaitly · Quoc V Le · Oriol Vinyals · Ilya Sutskever · David Sussillo · Samy Bengio -
2016 Poster: Exponential expressivity in deep neural networks through transient chaos »
Ben Poole · Subhaneil Lahiri · Maithra Raghu · Jascha Sohl-Dickstein · Surya Ganguli -
2016 Poster: An equivalence between high dimensional Bayes optimal inference and M-estimation »
Madhu Advani · Surya Ganguli -
2016 Poster: Deep Learning Models of the Retinal Response to Natural Scenes »
Lane McIntosh · Niru Maheswaranathan · Aran Nayebi · Surya Ganguli · Stephen Baccus -
2015 Poster: Deep Knowledge Tracing »
Chris Piech · Jonathan Bassen · Jonathan Huang · Surya Ganguli · Mehran Sahami · Leonidas Guibas · Jascha Sohl-Dickstein -
2014 Workshop: Deep Learning and Representation Learning »
Andrew Y Ng · Yoshua Bengio · Adam Coates · Roland Memisevic · Sharanyan Chetlur · Geoffrey E Hinton · Shamim Nemati · Bryan Catanzaro · Surya Ganguli · Herbert Jaeger · Phil Blunsom · Leon Bottou · Volodymyr Mnih · Chen-Yu Lee · Rich M Schwartz -
2014 Poster: Identifying and attacking the saddle point problem in high-dimensional non-convex optimization »
Yann N Dauphin · Razvan Pascanu · Caglar Gulcehre · Kyunghyun Cho · Surya Ganguli · Yoshua Bengio -
2013 Poster: A memory frontier for complex synapses »
Subhaneil Lahiri · Surya Ganguli -
2013 Oral: A memory frontier for complex synapses »
Subhaneil Lahiri · Surya Ganguli -
2010 Poster: Short-term memory in neuronal networks through dynamical compressed sensing »
Surya Ganguli · Haim Sompolinsky