Self-supervised Representation Learning from Random Data Projectors

Augmentation-based SSRL algorithms have pushed the boundaries of performance in computer vision and natural language processing, but are often not directly applicable to other data modalities, and can conflict with application-specific data augmentation constraints. We present an SSRL approach that can be applied to any data modality because it does not rely on augmentations. We show that high-quality data representations can be learned by reconstructing random data projections, and evaluate the proposed approach on a range of representation learning tasks that span diverse modalities and real-world applications.