Skip to yearly menu bar Skip to main content


Poster

Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization

Bruno Korbar · Du Tran · Lorenzo Torresani

Room 517 AB

Keywords: [ Unsupervised Learning ] [ Computer Vision ] [ Efficient Training Methods ] [ Speech Recognition ] [ Activity and Event Recognition ]


Abstract:

There is a natural correlation between the visual and auditive elements of a video. In this work we leverage this connection to learn general and effective models for both audio and video analysis from self-supervised temporal synchronization. We demonstrate that a calibrated curriculum learning scheme, a careful choice of negative examples, and the use of a contrastive loss are critical ingredients to obtain powerful multi-sensory representations from models optimized to discern temporal synchronization of audio-video pairs. Without further fine-tuning, the resulting audio features achieve performance superior or comparable to the state-of-the-art on established audio classification benchmarks (DCASE2014 and ESC-50). At the same time, our visual subnet provides a very effective initialization to improve the accuracy of video-based action recognition models: compared to learning from scratch, our self-supervised pretraining yields a remarkable gain of +19.9% in action recognition accuracy on UCF101 and a boost of +17.7% on HMDB51.

Live content is unavailable. Log in and register to view live content