Timezone: »

Invariance and identifiability issues for word embeddings
Rachel Carrington · Karthik Bharath · Simon Preston

Thu Dec 12 05:00 PM -- 07:00 PM (PST) @ East Exhibition Hall B + C #135

Word embeddings are commonly obtained as optimisers of a criterion function f of a text corpus, but assessed on word-task performance using a different evaluation function g of the test data. We contend that a possible source of disparity in performance on tasks is the incompatibility between classes of transformations that leave f and g invariant. In particular, word embeddings defined by f are not unique; they are defined only up to a class of transformations to which f is invariant, and this class is larger than the class to which g is invariant. One implication of this is that the apparent superiority of one word embedding over another, as measured by word task performance, may largely be a consequence of the arbitrary elements selected from the respective solution sets. We provide a formal treatment of the above identifiability issue, present some numerical examples, and discuss possible resolutions.

Author Information

Rachel Carrington (University of Nottingham)
Karthik Bharath (University of Nottingham)
Simon Preston (University of Nottingham)

More from the Same Authors