Estimating Brain Activity with High Spatial and Temporal Resolution using a Naturalistic MEG-fMRI Encoding Model

Current non-invasive neuroimaging techniques trade off between spatial resolution and temporal resolution. While magnetoencephalography (MEG) can capture rapid neural dynamics and functional magnetic resonance imaging (fMRI) can spatially localize brain activity, a unified picture that preserves both high resolutions remains elusive with existing source localization or MEG-fMRI fusion methods, especially for single-trial naturalistic data. We introduce a transformer-based encoding framework that combines MEG and fMRI from two naturalistic speech comprehension experiments with identical stimuli to estimate latent cortical source responses with high spatiotemporal resolution. Participants listened passively to more than 7h of narrative stories while whole-head MEG was recorded; each also completed a 3T fMRI scan on repeated presentations of one anchor story. For the remaining stories, we design a pipeline to project an open fMRI dataset collected on identical stimuli [LeBel et al., 2023] onto each participant’s cortical surface. We build a transformer-based encoding model whose latent layer represents our estimates of reconstructed cortical sources. A sliding‑window self‑attention enables the model to leverage the preceding 10s of stimulus features. The network is trained end‑to‑end with a joint MEG and fMRI loss. Our model's predictive performance is on par with single‑modality linear encoding models for held‑out stories, yet it also yields source estimates with high spatial and temporal resolution that outperform classic minimum norm solutions in simulation experiments. Furthermore, the model demonstrates strong generalizability across unseen participants and modalities, successfully performing zero-shot prediction of electrocorticography (ECoG) in an entirely new dataset. By integrating the power of large naturalistic experiments, MEG, and fMRI with a transformer-based encoding model, we propose a practical route to millisecond‑and‑millimeter brain mapping.