Timezone: »

 
Poster
Alignment Attention by Matching Key and Query Distributions
Shujian Zhang · Xinjie Fan · Huangjie Zheng · Korawat Tanwisuth · Mingyuan Zhou

Wed Dec 08 12:30 AM -- 02:00 AM (PST) @
in Poster Session 3 »

The neural attention mechanism has been incorporated into deep neural networks to achieve state-of-the-art performance in various domains. Most such models use multi-head self-attention which is appealing for the ability to attend to information from different perspectives. This paper introduces alignment attention that explicitly encourages self-attention to match the distributions of the key and query within each head. The resulting alignment attention networks can be optimized as an unsupervised regularization in the existing attention framework. It is simple to convert any models with self-attention, including pre-trained ones, to the proposed alignment attention. On a variety of language understanding tasks, we show the effectiveness of our method in accuracy, uncertainty estimation, generalization across domains, and robustness to adversarial attacks. We further demonstrate the general applicability of our approach on graph attention and visual question answering, showing the great potential of incorporating our alignment method into various attention-related tasks.

Keywords: Deep Learning · Robustness · Adversarial Robustness and Security · Vision · Graph Learning · Language

Author Information

Shujian Zhang (UT Austin)
Xinjie Fan (UT Austin)
Huangjie Zheng (University of Texas, Austin)
Korawat Tanwisuth (The University of Texas at Austin)
Mingyuan Zhou (University of Texas at Austin)

More from the Same Authors