Timezone: »

 
Poster
KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation
Ta-Chung Chi · Ting-Han Fan · Peter J. Ramadge · Alexander Rudnicky

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #440
in Poster Session 6 »

Relative positional embeddings (RPE) have received considerable attention since RPEs effectively model the relative distance among tokens and enable length extrapolation. We propose KERPLE, a framework that generalizes relative position embedding for extrapolation by kernelizing positional differences. We achieve this goal using conditionally positive definite (CPD) kernels, a class of functions known for generalizing distance metrics. To maintain the inner product interpretation of self-attention, we show that a CPD kernel can be transformed into a PD kernel by adding a constant offset. This offset is implicitly absorbed in the Softmax normalization during self-attention. The diversity of CPD kernels allows us to derive various RPEs that enable length extrapolation in a principled way. Experiments demonstrate that the logarithmic variant achieves excellent extrapolation performance on three large language modeling datasets. Our implementation and pretrained checkpoints are released at~\url{https://github.com/chijames/KERPLE.git}.

Keywords: kernel method · Length Extrapolation · Transformer Language Modeling

Author Information

Ta-Chung Chi (Carnegie Mellon University)

I am a 4th-year PhD student at the Language Technologies Institute in the School of Computer Science at Carnegie Mellon University advised by professor Alexander I. Rudnicky. My research interests lie in the field of dialogue system and related NLP topics.

Ting-Han Fan (Princeton University)
Peter J. Ramadge (Princeton)
Alexander Rudnicky (Carnegie Mellon University)

Alexander I. Rudnicky is Professor Emeritus in the Language Technologies Institute in the School of Computer Science at Carnegie Mellon University. Dr. Rudnicky's research has spanned many aspects of spoken language, including language modeling, spoken language system architectures, multi-modal interaction, and the analysis of conversational structure. Dr. Rudnicky and his students developed the PocketSphinx recognition system and the Ravenclaw dialog manager. More recently, Dr. Rudnicky has been active in research on open-domain conversational systems. Dr. Rudnicky interests in learning include induction of concepts and task structure from conversation, and the design of intelligent systems that proactively seek to acquire knowledge from people.

More from the Same Authors