SO(3)-Equivariant Representation Learning in 2D Images

Imaging physical objects that are free to rotate and translate in 3D is challenging. Whilean object’s pose and location do not change its nature, varying them presents problemsfor current vision models. Equivariant models account for these nuisance transformations,but current architectures only model either 2D transformations of 2D signals or 3D trans-formations of 3D signals. Here, we propose a novel convolutional layer consisting of 2Dprojections of 3D filters that models 3D equivariances of 2D signals—critical for capturingthe full space of spatial transformations of objects in imaging domains such as cryo-EM. Weadditionally present methods for aggregating our rotation-specific outputs. We demonstrate improvement on several tasks, including particle picking and pose estimation.