MLINI - 2nd NIPS Workshop on Machine Learning and Interpretation in Neuroimaging (2 day)

1.) Aim

We propose a two day workshop on the topic of machine learning approaches in neuroscience and neuroimaging. We believe that both machine learning and neuroimaging can learn from each other as the two communities overlap and enter an intense exchange of ideas and research questions. Methodological developments in machine learning spurn novel paradigms in neuroimaging, neuroscience motivates methodological advances in computational analysis. In this context many controversies and open questions exist. The goal of the workshop is to pinpoint these issues, sketch future directions, and tackle open questions in the light of novel methodology.

The first workshop of this series at NIPS 2011 built upon earlier events in 2006 and 2008. Last year’s workshop included many invited speakers, and was centered around two panel discussions, during which 2 questions were discussed: the interpretability of machine learning findings, and the shift of paradigms in the neuroscience community. The discussion was inspiring, and made clear, that there is a tremendous amount the two communities can learn from each other benefiting from communication across the disciplines.

The aim of the workshop is to offer a forum for the overlap of these communities. Besides interpretation, and the shift of paradigms, many open questions remain. Among them:

• How suitable are MVPA and inference methods for brain mapping?
• How can we assess the specificity and sensitivity?
• What is the role of decoding vs. embedded or separate feature selection?
• How can we use these approaches for a flexible and useful representation of neuroimaging data?
• What can we accomplish with generative vs. discriminative modelling?
• Can and should the Machine Learning community provide a standard repertoire of methods for the Neuroimaging community to use (e.g. in choosing a classifier)?

2.) Background

Modern multivariate statistical methods have been increasingly applied to various problems in neuroimaging, including “mind reading”, “brain mapping”, clinical diagnosis and prognosis. Multivariate pattern analysis (MVPA) methods are designed to examine complex relationships between large-dimensional signals, such as brain MRI images, and an outcome of interest, such as the category of a stimulus, with a limited amount of data. The MVPA approach is in contrast with the classical mass-univariate (MUV) approach that treats each individual imaging measurement in isolation.

While MUV is useful in localizing effects characterized by localized activity of individual regions, i.e., brain mapping, it is less suited to constructing models that can make prediction at the subject level. Even more importantly, MUV ignores relationships between disjoint anatomical regions, while a growing body of neuroscientific evidence is pointing to an organization of the brain that is comprised of large-scale, distributed networks. These networks exhibit coherent functional activity and can be targeted by disease, resulting in correlations in atrophy. By examining the entire image pattern of both functional and structural data, rather than voxel-level measurements, MVPA offers a unique opportunity to examine and reveal these network-level associations.

Yet, as a new approach in neuroimaging, MVPA is surrounded with unresolved, controversial issues.

In this workshop, we intend to investigate the implications that follow from adopting machine-learning methods for studying brain function. In particular, this concerns the question how these methods may be used to represent cognitive states, and what ramifications this has for consequent theories of cognition. Besides providing a rationale for the use of machine-learning methods in studying brain function, a further goal of this workshop is to identify shortcomings of state-of-the-art approaches and initiate research efforts that increase the impact of machine learning on cognitive neuroscience.

Moreover, from the machine learning perspective, neuroimaging is a rich source of challenging problems that can facilitate development of novel approaches; for example, feature extraction and feature selection is particularly important since the primary objective machine learning analysis of neuroimaging data is to gain a scientific insight rather than simply learn a ``black-box'' predictor. However, unlike some other applications where the set features might be quite well-explored and established by now, neuroimaging is a domain where a machine-learning researcher cannot simply "ask a domain expert what features should be used", since this is essentially the question the domain expert themselves are trying to figure out. While the current neuroscientific knowledge can guide the definition of specialized 'brain areas', more complex patterns of brain activity, such as spatio-temporal patterns, functional network patterns, and other multivariate dependencies remain to be discovered mainly via statistical analysis.

3.) Open questions and possible topics for contributions

I. Machine learning and pattern recognition methodology

1. Statistics. The common approach to quantify the model fit in MVPA methods is via metrics like Area Under the ROC curve, average accuracy, and mean square error obtained from cross-validation. However, we are also interested in other statistical quantities: e.g. confidence intervals and statistical significance of our estimates, the detected regions, and their relationship to the experimental conditions. What are methods that achieve statistical interpretability of observation made via MVPA approaches?

2. Generative modeling versus Discriminative modeling. What are appropriate approaches for specific problems, what questions can be posed in either of these frameworks, what are overlapping, what are complementary characteristics - potentially depending on the specific neuroscientific question?

3. Embedded vs. separate feature selection and decoding. Several recent approaches perform both feature selection, and decoding or classification. Can, or should they be decoupled, and which considerations are important for each choice?

II. Causal inference and interpretability in neuroimaging

1. Biological interpretability. Multivariate models are, by construction, difficult to interpret and visualize since they are based on patterns that span the image and are not localized. Furthermore, non-linear models, such as those used in kernel-based methods, are even harder to characterize since they cannot be represented with a single “discriminative” map.

2. True specificity and sensitivity in the general population. Traditional computational anatomy studies compare cases and controls to characterize effects. MVPA methods offer the ability to examine multi-variate effects and make accurate subject-level predictions. The traditional paradigm of cross-validation on case-control data, however, is likely to over-estimate the accuracy of MVPA methods. It is not clear how these models will perform in the general population, where we have heterogeneity in normals and many other “similar” disease conditions to consider – e.g. Alzheimer’s and other dementia types.

3. How to deal with confounding factors such as age, gender, subject motion, etc? Do we pre-process the data to regress out these effects or include them in the MVPA model? How do we combine features with different units in the MVPA model?

III. Linking machine learning, neuroimaging and neuroscience

1. How suitable are MVPA methods for brain mapping? Brain mapping deals with the problem of localizing regions that are recruited for certain functional tasks, such as viewing a stimulus. Is the use of MVPA methods for brain mapping appropriate? Isn’t it true that a sophisticated MVPA model, given enough data, is likely to be able to discriminate between stimuli with an accuracy significantly better than chance across signals from all brain regions? There is a basic difference between a brain region “encoding” for a stimulus and it being “tuned” to a stimulus. How far can the MVPA framework go in helping to understand the functional specificity of brain regions?

2. Flexible representation of functional and anatomical neuroimaging data. The possibilities of representing data offered by machine learning approaches such as manifold learning are currently only partially used. We would like to understand how machine learning and the mapping approaches it offers can be used to better understand neuroimaging data in both exploratory research, and the clinical setting?

3. Model based methods and their link to neuroscience. Model based methods are only slowly adopted in the neuroscience community, with the literature being dominated by standard MUV approaches. There seems to be a reservation regarding the generalization power of and their verification in experiments. What is the reason for this reservation? What are the true vs. the perceived limitations of MVPA? How can we perform model selection in the light of bias and generalization performance.