Timezone: »

 
Poster
Relational Self-Attention: What's Missing in Attention for Video Understanding
Manjin Kim · Heeseung Kwon · CHUNYU WANG · Suha Kwak · Minsu Cho

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @
in Poster Session 1 »

Convolution has been arguably the most important feature transform for modern neural networks, leading to the advance of deep learning. Recent emergence of Transformer networks, which replace convolution layers with self-attention blocks, has revealed the limitation of stationary convolution kernels and opened the door to the era of dynamic feature transforms. The existing dynamic transforms, including self-attention, however, are all limited for video understanding where correspondence relations in space and time, i.e., motion information, are crucial for effective representation. In this work, we introduce a relational feature transform, dubbed the relational self-attention (RSA), that leverages rich structures of spatio-temporal relations in videos by dynamically generating relational kernels and aggregating relational contexts. Our experiments and ablation studies show that the RSA network substantially outperforms convolution and self-attention counterparts, achieving the state of the art on the standard motion-centric benchmarks for video action recognition, such as Something-Something-V1&V2, Diving48, and FineGym.

Keywords: Deep Learning · Transformers

Author Information

Manjin Kim (POSTECH)
Heeseung Kwon (POSTECH)
CHUNYU WANG (Peking University)
Suha Kwak (POSTECH)
Minsu Cho (POSTECH)

More from the Same Authors