One Question at a Time: A Semantic Bottleneck for Interpretable Visual Brain Decoding from fMRI

Decoding of visual stimuli from noninvasive neuroimaging techniques such as functional magnetic resonance has advanced rapidly in the last few years; yet, most high-performing brain decoding models rely on complicated, non-intelligible latent spaces (e.g., CLIP).In this study, we present an interpretable brain decoding framework; our key innovation is the insertion of a semantic bottleneck into BrainDiffuser, a well established brain decoding pipeline. Specifically, we build a $214-\text{dimensional}$ binary interpretable space $\mathcal{L}$ for images, in which each dimension answers a specific question about the image (e.g., "Is there a person?", "Is it outdoors?"). A first ridge regression maps the activity of the voxel to this semantic space. Because this mapping is linear, its weight matrix can be visualized as maps of voxel importance for each dimension of $\mathcal{L}$, revealing which cortical regions most influence each semantic dimension. A second regression then transforms these concept vectors into CLIP embeddings required to produce the final decoded image, conditioning the BrainDiffuser model. We find that voxel-wise weight maps for individual questions are highly consistent with canonical category-selective regions in the visual cortex (face, bodies, places, words), simultaneously revealing that activation distributions, not merely location, bear semantic meaning in the brain. Visual brain decoding performances are only slightly lower compared to the original BrainDiffuser metrics (e.g., CLIP similarity is decreased by $\leq 4\%$), but offering substantial gains in interpretability and neuroscientific insights. These results show that our interpretable brain decoding pipeline enables voxel-level analysis of semantic representations in the human brain without sacrificing decoding accuracy.