Timezone: »

 
Poster
Maximum a posteriori natural scene reconstruction from retinal ganglion cells with deep denoiser priors
Eric Wu · Nora Brackbill · Alexander Sher · Alan Litke · Eero Simoncelli · E.J. Chichilnisky

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #624
in Poster Session 5 »

Visual information arriving at the retina is transmitted to the brain by signals in the optic nerve, and the brain must rely solely on these signals to make inferences about the visual world. Previous work has probed the content of these signals by directly reconstructing images from retinal activity using linear regression or nonlinear regression with neural networks. Maximum a posteriori (MAP) reconstruction using retinal encoding models and separately-trained natural image priors offers a more general and principled approach. We develop a novel method for approximate MAP reconstruction that combines a generalized linear model for retinal responses to light, including their dependence on spike history and spikes of neighboring cells, with the image prior implicitly embedded in a deep convolutional neural network trained for image denoising. We use this method to reconstruct natural images from ex vivo simultaneously-recorded spikes of hundreds of retinal ganglion cells uniformly sampling a region of the retina. The method produces reconstructions that match or exceed the state-of-the-art in perceptual similarity and exhibit additional fine detail, while using substantially fewer model parameters than previous approaches. The use of more rudimentary encoding models (a linear-nonlinear-Poisson cascade) or image priors (a 1/f spectral model) significantly reduces reconstruction performance, indicating the essential role of both components in achieving high-quality reconstructed images from the retinal signal.

Keywords: Neuroscience · Neural Coding · retina · encoding model · image reconstruction · ganglion cell · Plug and Play · natural scenes · neural decoding · image prior
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Eric Wu (Stanford University)
Nora Brackbill (Stanford University)
Alexander Sher (Santa Cruz Institute for Particle Physics, University of California, Santa Cruz)
Alan Litke (Santa Cruz Institute for Particle Physics, University of California, Santa Cruz)
Eero Simoncelli (FlatIron Institute / New York University)

Eero P. Simoncelli received the B.S. degree in Physics in 1984 from Harvard University, studied applied mathematics at Cambridge University for a year and a half, and then received the M.S. degree in 1988 and the Ph.D. degree in 1993, both in Electrical Engineering from the Massachusetts Institute of Technology. He was an Assistant Professor in the Computer and Information Science department at the University of Pennsylvania from 1993 until 1996. He moved to New York University in September of 1996, where he is currently a Professor in Neural Science, Mathematics, and Psychology. In August 2000, he became an Associate Investigator of the Howard Hughes Medical Institute, under their new program in Computational Biology. In Fall 2020, he resigned his HHMI appointment to become the scientific director of the Center for Computational Neuroscience at the Flatiron Institute, of the Simons Foundation. His research interests span a wide range of topics in the representation and analysis of visual images, in both machine and biological systems.

E.J. Chichilnisky (Stanford University)

More from the Same Authors