Optimizing fMRI Data Acquisition for Decoding Natural Speech with Limited Participants

We present a systematic investigation into decoding perceived natural speech from fMRI data in a participant-limited setting. Using a publicly available dataset of eight participants (LeBel et al., 2023), we demonstrate that deep neural networks trained with a contrastive objective can effectively decode unseen natural speech by retrieving the embedding of perceived sentences from fMRI activity. We found that decoding performance directly correlates with the amount of training data available per participant. In this data regime, multi-subject training does not improve decoding accuracy compared to the single-subject approach. Additionally, training on similar or different stimuli across subjects has a negligible effect on decoding accuracy. Finally, we find that our decoders model both syntactic and semantic features, and that stories containing sentences with complex syntax or rich semantic content are more challenging to decode. While our results demonstrate the benefits of having extensive data per participant (deep phenotyping), they suggest that leveraging multi-subject data for natural speech decoding likely requires deeper phenotyping or a substantially larger cohort.