A model of contextual representations and their role for linguistic prediction

The predictability value of a word corresponds to its frequency of use during a cloze task, and it has been correlated with processing time and also with the N400 component in EEG studies. Most importantly, the predictability value has been used extensively in the literature, even though we still know almost nothing about how a cloze task is performed.Using an interdisciplinary perspective regarding the nature of linguistic prediction and the kinds of cognitive processes involved therein, I developed a new theoretically-driven computational approach that revisits the derivation of the predictability score. Empirical results in psycholinguistics and neurolinguistics do not support the Strong Prediction View, and they tend to show that semantics and syntax are processed independently, and that the semantic stream has precedence over the syntactic stream. In this poster, I present a model of linguistic prediction that is compatible with these results in which I differentiate between the contribution coming from different levels of semantic granularity and the one coming from the coordination aspect of linguistic interaction.In this model, a linguistic prediction is derived from the combination of the contributions from four kinds of sentence-level representations. Each kind of representation triggers an activation signal that spreads throughout a conceptual space where the level of activation of any concepts at a particular time represents the degree by which they are triggered by the information retrieved from the truncated sentence and the global context. These conceptual spaces are derived from similarity spaces obtained from pre-trained word embeddings. To represent these four sentence-level representations, I use the Learning and Inference with Schemas and Analogy (LISA) approach which is a hybrid symbolic-connectionist model that codes relational structure and can represent both objects and relational roles as patterns of activation over units representing semantic features.When assigning a relative probability of occurrence for potential continuations, I considered both the contribution from the truncated sentence and the contribution coming from two kinds of contextual information: a topic model and a situation model. The topic model is derived from a pre-trained topic distribution space representing the relationship between topics and words, and the situation model is derived by combining the four kinds of sentence-level representations. These contextual representations are derived from the bottom-up from the meaning expressed at the sentence level, and they, in turn, influence the predictive process by constraining the linguistic prediction via a top-down signal. I then present a multi-layered processing structure of linguistic prediction that integrates the contribution from the sentence-level representations, the contribution from the contextual level representations, and the constant interaction between the two. Preliminary empirical adequacy was assessed by three worked-out examples (high-constraining sentence, low-constraining sentence, sentence with prior discourse context) for which the theory matches the ordering that was obtained empirically.This model of linguistic prediction illustrates the crucial connection between the representational levels involved in pragmatic processing, and it conceptualizes the pragmatic stream as a processing structure. This view is compatible with recent hierarchical models of linguistic processing, and it shares some features with computationally explicit connectionist accounts of the prediction process.