Timezone: »

Visualizing and Measuring the Geometry of BERT
Emily Reif · Ann Yuan · Martin Wattenberg · Fernanda B Viegas · Andy Coenen · Adam Pearce · Been Kim

Tue Dec 10 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #172

Transformer architectures show significant promise for natural language processing. Given that a single pretrained model can be fine-tuned to perform well on many different tasks, these networks appear to extract generally useful linguistic features. A natural question is how such networks represent this information internally. This paper describes qualitative and quantitative investigations of one particularly effective model, BERT. At a high level, linguistic features seem to be represented in separate semantic and syntactic subspaces. We find evidence of a fine-grained geometric representation of word senses. We also present empirical descriptions of syntactic representations in both attention matrices and individual word embeddings, as well as a mathematical argument to explain the geometry of these representations.

Author Information

Emily Reif (Google)
Ann Yuan (Google)
Martin Wattenberg (Google)

Fernanda Viégas and Martin Wattenberg co-lead Google’s PAIR (People+AI Research) initiative, part of Google Brain. Their work in machine learning focuses on transparency and interpretability, as part of a broad agenda to improve human/AI interaction. They are well known for their contributions to social and collaborative visualization, and the systems they’ve created are used daily by millions of people. Their visualization-based artwork has been exhibited worldwide, and is part of the permanent collection of Museum of Modern Art in New York.

Fernanda B Viegas (Google)
Andy Coenen (Google)
Adam Pearce (Google)
Been Kim (Google)

More from the Same Authors