Skip to yearly menu bar Skip to main content

Workshop: Machine Learning for Audio

Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localization

Edward Fish · Jon Weinbren · Andrew Gilbert


Temporal Action Localization (TAL) aims to identify actions' start, end, and class labels in untrimmed videos. While recent advancements using transformer networks and Temporal Pyramid Networks (TPN) have enhanced visual feature recognition in TAL tasks, there's an under-explored area of integrating multi-resolution audio features into such frameworks. This paper introduces Multi-Resolution Audio-Visual Feature Fusion (MRAV-FF), an innovative method to merge audio-visual data across different temporal resolutions. Central to our approach is a hierarchical gated cross-attention mechanism, which discerningly weighs the importance of audio information at diverse temporal scales. Such a technique not only refines the precision of regression boundaries but also bolsters classification accuracy. Importantly, MRAV-FF is versatile, making it compatible with existing TPN TAL architectures and offering a significant enhancement in performance when audio data is available.

Chat is not available.