Timezone: »

 
Poster
Visualizing and Measuring the Geometry of BERT
Emily Reif · Ann Yuan · Martin Wattenberg · Fernanda Viegas · Andy Coenen · Adam Pearce · Been Kim

Tue Dec 10 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #172
in Deep Learning -- Visualization or Exposition Techniques for Deep Networks »

Transformer architectures show significant promise for natural language processing. Given that a single pretrained model can be fine-tuned to perform well on many different tasks, these networks appear to extract generally useful linguistic features. A natural question is how such networks represent this information internally. This paper describes qualitative and quantitative investigations of one particularly effective model, BERT. At a high level, linguistic features seem to be represented in separate semantic and syntactic subspaces. We find evidence of a fine-grained geometric representation of word senses. We also present empirical descriptions of syntactic representations in both attention matrices and individual word embeddings, as well as a mathematical argument to explain the geometry of these representations.

Keywords: Attention Models · Embedding Approaches · Deep Learning
[ Paper]  [ Slides] 

Author Information

Emily Reif (Google)
Ann Yuan (Google)
Martin Wattenberg (Google)

Fernanda Viégas and Martin Wattenberg co-lead Google’s PAIR (People+AI Research) initiative, part of Google Brain. Their work in machine learning focuses on transparency and interpretability, as part of a broad agenda to improve human/AI interaction. They are well known for their contributions to social and collaborative visualization, and the systems they’ve created are used daily by millions of people. Their visualization-based artwork has been exhibited worldwide, and is part of the permanent collection of Museum of Modern Art in New York.

Fernanda Viegas (Google)
Andy Coenen (Google)
Adam Pearce (Google)
Been Kim (Google)

More from the Same Authors