Timezone: »
Humans are successfully able to recognize objects in a variety of image distributions. Today's artificial neural networks (ANNs), on the other hand, struggle to recognize objects in many image domains, especially those different from the training distribution. It is currently unclear which parts of the ANNs could be improved in order to close this generalization gap. In this work, we used recordings from primate high-level visual cortex (IT) to isolate whether ANNs lag behind primate generalization capabilities because of their encoder (transformations up to the penultimate layer), or their decoder (linear transformation into class labels). Specifically, we fit a linear decoder on images from one domain and evaluate transfer performance on twelve held-out domains, comparing fitting on primate IT representations vs. representations in ANN penultimate layers. To fairly compare, we scale the number of each ANN's units so that its in-domain performance matches that of the sampled IT population (i.e. 71 IT neural sites, 73% binary-choice accuracy). We find that the sampled primate population achieves, on average, 68% performance on the held-out-domains. Comparably sampled populations from ANN model units generalize less well, maintaining on average 60%. This is independent of the number of sampled units: models' out-of-domain accuracies consistently lag behind primate IT. These results suggest that making ANN model units more like primate IT will improve the generalization performance of ANNs.
Author Information
Marliawaty I Gusti Bagus (MIT)
Tiago Marques (MIT)
Sachi Sanghavi (University of Wisconsin–Madison)
James J DiCarlo (Massachusetts Institute of Technology)
Prof. DiCarlo received his Ph.D. in biomedical engineering and his M.D. from Johns Hopkins in 1998, and did his postdoctoral training in primate visual neurophysiology at Baylor College of Medicine. He joined the MIT faculty in 2002. He is a Sloan Fellow, a Pew Scholar, and a McKnight Scholar. His lab’s research goal is a computational understanding of the brain mechanisms that underlie object recognition. They use large-scale neurophysiology, brain imaging, optogenetic methods, and high-throughput computational simulations to understand how the primate ventral visual stream is able to untangle object identity from other latent image variables such as object position, scale, and pose. They have shown that populations of neurons at the highest cortical visual processing stage (IT) rapidly convey explicit representations of object identity, and that this ability is reshaped by natural visual experience. They have also shown how visual recognition tests can be used to discover new, high-performing bio-inspired algorithms. This understanding may inspire new machine vision systems, new neural prosthetics, and a foundation for understanding how high-level visual representation is altered in conditions such as agnosia, autism and dyslexia.
Martin Schrimpf (MIT)
More from the Same Authors
-
2021 : ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation »
Chuang Gan · Jeremy Schwartz · Seth Alter · Damian Mrowca · Martin Schrimpf · James Traer · Julian De Freitas · Jonas Kubilius · Abhishek Bhandwaldar · Nick Haber · Megumi Sano · Kuno Kim · Elias Wang · Michael Lingelbach · Aidan Curtis · Kevin Feigelis · Daniel Bear · Dan Gutfreund · David Cox · Antonio Torralba · James J DiCarlo · Josh Tenenbaum · Josh McDermott · Dan Yamins -
2022 : Measuring the Alignment of ANNs and Primate V1 on Luminance and Contrast Response Characteristics »
Stephanie Olaiya · Tiago Marques · James J DiCarlo -
2022 : Implementing Divisive Normalization in CNNs Improves Robustness to Common Image Corruptions »
Andrew Cirincione · Reginald Verrier · Artiom Bic · Stephanie Olaiya · James J DiCarlo · Lawrence Udeigwe · Tiago Marques -
2023 Poster: Robustified ANNs Reveal Wormholes Between Human Category Percepts »
Guy Gaziv · Michael Lee · James J DiCarlo -
2022 : A report on recent experimental tests of two predictions of contemporary computable models of the biological deep neural network underling primate visual intelligence »
James J DiCarlo -
2022 Poster: How Well Do Unsupervised Learning Algorithms Model Human Real-time and Life-long Learning? »
Chengxu Zhuang · Ziyu Xiang · Yoon Bai · Xiaoxuan Jia · Nicholas Turk-Browne · Kenneth Norman · James J DiCarlo · Dan Yamins -
2021 : Combining Different V1 Brain Model Variants to Improve Robustness to Image Corruptions in CNNs »
Avinash Baidya · Joel Dapello · James J DiCarlo · Tiago Marques -
2021 Poster: Neural Population Geometry Reveals the Role of Stochasticity in Robust Perception »
Joel Dapello · Jenelle Feather · Hang Le · Tiago Marques · David Cox · Josh McDermott · James J DiCarlo · Sueyeon Chung -
2021 : ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation »
Chuang Gan · Jeremy Schwartz · Seth Alter · Damian Mrowca · Martin Schrimpf · James Traer · Julian De Freitas · Jonas Kubilius · Abhishek Bhandwaldar · Nick Haber · Megumi Sano · Kuno Kim · Elias Wang · Michael Lingelbach · Aidan Curtis · Kevin Feigelis · Daniel Bear · Dan Gutfreund · David Cox · Antonio Torralba · James J DiCarlo · Josh Tenenbaum · Josh McDermott · Dan Yamins -
2020 Poster: Simulating a Primary Visual Cortex at the Front of CNNs Improves Robustness to Image Perturbations »
Joel Dapello · Tiago Marques · Martin Schrimpf · Franziska Geiger · David Cox · James J DiCarlo -
2020 Spotlight: Simulating a Primary Visual Cortex at the Front of CNNs Improves Robustness to Image Perturbations »
Joel Dapello · Tiago Marques · Martin Schrimpf · Franziska Geiger · David Cox · James J DiCarlo -
2019 Poster: Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs »
Jonas Kubilius · Martin Schrimpf · Kohitij Kar · Rishi Rajalingham · Ha Hong · Najib Majaj · Elias Issa · Pouya Bashivan · Jonathan Prescott-Roy · Kailyn Schmidt · Aran Nayebi · Daniel Bear · Daniel Yamins · James J DiCarlo -
2019 Oral: Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs »
Jonas Kubilius · Martin Schrimpf · Ha Hong · Najib Majaj · Rishi Rajalingham · Elias Issa · Kohitij Kar · Pouya Bashivan · Jonathan Prescott-Roy · Kailyn Schmidt · Aran Nayebi · Daniel Bear · Daniel Yamins · James J DiCarlo -
2018 Poster: Task-Driven Convolutional Recurrent Models of the Visual System »
Aran Nayebi · Daniel Bear · Jonas Kubilius · Kohitij Kar · Surya Ganguli · David Sussillo · James J DiCarlo · Daniel Yamins -
2017 : Panel on "What neural systems can teach us about building better machine learning systems" »
Timothy Lillicrap · James J DiCarlo · Christopher Rozell · Viren Jain · Nathan Kutz · William Gray Roncal · Bingni Brunton -
2017 : Can brain data be used to reverse engineer the algorithms of human perception? »
James J DiCarlo -
2013 Poster: Hierarchical Modular Optimization of Convolutional Networks Achieves Representations Similar to Macaque IT and Human Ventral Stream »
Daniel L Yamins · Ha Hong · Charles Cadieu · James J DiCarlo -
2013 Tutorial: Mechanisms Underlying Visual Object Recognition: Humans vs. Neurons vs. Machines »
James J DiCarlo