Decoding and Reconstructing Visual Experience from Brain Activity with Generative Latent Representations

The brain's bidirectional processing is thought to parallel the functions of recognition (bottom-up) and generative (top-down) models in AI. While prior work has established a hierarchical correspondence between the visual cortex and recognition models, whether generative models exhibit a similar neural alignment remains an open question. To investigate this, we present a unified framework that decodes fMRI signals into the latent representations of both model classes for visual reconstruction. Our analysis of perception data revealed that the generative model achieved decoding and reconstruction performance comparable to the recognition model, though its hierarchical correspondence with the visual cortex was weaker and followed a different trend. An application to imagery data showed low decoding accuracies but a different pattern from perception. Our work contributes a comparative pipeline for studying generative representations and provides a framework for future investigations into the brain's bidirectional processing architecture.