Demo: Real-time reconstruction of human visual perception from fMRI

We demonstrate a first-of-its-kind pipeline that reconstructs perceived images from fMRI brain activity in real-time. This live demonstration showcases how our open-source framework, RT-Cloud, can enable state-of-the-art AI workflows for real-time fMRI at any standard MRI facility.

Real-time fMRI analysis has tremendous utility for both clinical and scientific applications; for example, it can support treatment of clinical conditions like depression and it can facilitate probing of fundamental learning mechanisms in the brain. However, existing real-time fMRI pipelines have focused on relatively simple forms of data analysis (e.g., multivariate decoding with a small number of stimulus classes). These simple forms of analysis provide a limited window into the richness of people’s ongoing thought. If we were able to decode fine-grained cognitive representations (e.g., reconstructing what a person is perceiving), this would substantially increase the utility of real-time fMRI as a brain-computer interface. However, extant pipelines for this type of decoding typically require processing times spanning hours or days and prohibitive amounts of training data per participant.

In this demonstration, we will show how we were able to overcome these challenges to perform the first ever real-time reconstruction of visual perception in 3 Tesla (3T) fMRI, a platform available at over 20,000 sites worldwide. We use RT-Cloud, a scalable open-source framework that enables real-time streaming and analysis of fMRI data, in conjunction with MindEye2, a state-of-the-art machine learning model for image reconstruction and retrieval. Prior work relied on extensive processing of the Natural Scenes Dataset (NSD), which collected 30-40 hours of data from each of a few participants. Additionally, NSD used 7 Tesla (7T) MRI, which is available at only about 100 sites around the world. Our pipeline involves pre-training MindEye2 on data from NSD and then fine-tuning on just 2-3 hours of 3T data from a new participant. After this, the model can support reconstruction of images viewed by that participant using fully real-time-compatible preprocessing. We adapt key steps from GLMsingle, a computationally-intensive offline preprocessing package, to fit within the computational envelope of real-time analysis.

Going forward, our real-time visual decoding pipeline can potentially support a range of novel applications (e.g., neurofeedback studies where participants are shown how their perception of an image differs from the ground truth image).

During the live workshop demonstration, we will reproduce our analysis, simulating data streaming (previously used in true real-time) as if we were receiving the data second-by-second. All of our code for simulation and real-time analysis with RT-Cloud is fully open-source and well-documented. Attendees will be able to explore our code and run our easy-to-replicate real-time analyses using real fMRI data.

We invite attendees interested in this project to connect with us on the MedARC Discord server (https://discord.gg/tVR4TWnRM9), an open science research forum operated by Sophont. We are actively seeking collaborators to help extend this framework to new and exciting real-time applications.

Funding was provided by the Princeton University Dean for Research Innovation Fund for New Industrial Collaborations, Sophont, and NIH grant RF1MH125318.