Timezone: »

Relaxing Equivariance Constraints with Non-stationary Continuous Filters
Tycho van der Ouderaa · David W. Romero · Mark van der Wilk

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #306

Equivariances provide useful inductive biases in neural network modeling, with the translation equivariance of convolutional neural networks being a canonical example. Equivariances can be embedded in architectures through weight-sharing and place symmetry constraints on the functions a neural network can represent. The type of symmetry is typically fixed and has to be chosen in advance. Although some tasks are inherently equivariant, many tasks do not strictly follow such symmetries. In such cases, equivariance constraints can be overly restrictive. In this work, we propose a parameter-efficient relaxation of equivariance that can effectively interpolate between a (i) non-equivariant linear product, (ii) a strict-equivariant convolution, and (iii) a strictly-invariant mapping. The proposed parameterisation can be thought of as a building block to allow adjustable symmetry structure in neural networks. In addition, we demonstrate that the amount of equivariance can be learned from the training data using backpropagation. Gradient-based learning of equivariance achieves similar or improved performance compared to the best value found by cross-validation and outperforms baselines with partial or strict equivariance on CIFAR-10 and CIFAR-100 image classification tasks.

Author Information

Tycho van der Ouderaa (Imperial College London)

The main topic of my PhD is learning structure and inductive biases in neural networks. The focus has been on learning symmetry from data, such as equivariance and invariance. The aim of the research is to make learning inductive biases and structure in machine learning models as easy as learning the weights.

David W. Romero (Vrije Universiteit Amsterdam)
Mark van der Wilk (Imperial College London)

More from the Same Authors