Timezone: »
Equivariances provide useful inductive biases in neural network modeling, with the translation equivariance of convolutional neural networks being a canonical example. Equivariances can be embedded in architectures through weight-sharing and place symmetry constraints on the functions a neural network can represent. The type of symmetry is typically fixed and has to be chosen in advance. Although some tasks are inherently equivariant, many tasks do not strictly follow such symmetries. In such cases, equivariance constraints can be overly restrictive. In this work, we propose a parameter-efficient relaxation of equivariance that can effectively interpolate between a (i) non-equivariant linear product, (ii) a strict-equivariant convolution, and (iii) a strictly-invariant mapping. The proposed parameterisation can be thought of as a building block to allow adjustable symmetry structure in neural networks. In addition, we demonstrate that the amount of equivariance can be learned from the training data using backpropagation. Gradient-based learning of equivariance achieves similar or improved performance compared to the best value found by cross-validation and outperforms baselines with partial or strict equivariance on CIFAR-10 and CIFAR-100 image classification tasks.
Author Information
Tycho van der Ouderaa (Imperial College London)
The main topic of my PhD is learning structure and inductive biases in neural networks. The focus has been on learning symmetry from data, such as equivariance and invariance. The aim of the research is to make learning inductive biases and structure in machine learning models as easy as learning the weights.
David W. Romero (Vrije Universiteit Amsterdam)
Mark van der Wilk (Imperial College London)
More from the Same Authors
-
2022 : Actually Sparse Variational Gaussian Processes »
Jake Cunningham · So Takao · Mark van der Wilk · Marc Deisenroth -
2022 : Recommendations for Baselines and Benchmarking Approximate Gaussian Processes »
Sebastian Ober · David Burt · Artem Artemev · Mark van der Wilk -
2022 : Sparse Convolutions on Lie Groups »
Tycho van der Ouderaa · Mark van der Wilk -
2022 : Causal Discovery using Marginal Likelihood »
Anish Dhir · Mark van der Wilk -
2022 Poster: Learning Partial Equivariances From Data »
David W. Romero · Suhas Lohit -
2022 Poster: Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations »
Alexander Immer · Tycho van der Ouderaa · Gunnar Rätsch · Vincent Fortuin · Mark van der Wilk -
2022 Poster: SnAKe: Bayesian Optimization with Pathwise Exploration »
Jose Pablo Folch · Shiqiang Zhang · Robert Lee · Behrang Shafei · David Walz · Calvin Tsay · Mark van der Wilk · Ruth Misener -
2022 Poster: Memory safe computations with XLA compiler »
Artem Artemev · Yuze An · Tilman Roeder · Mark van der Wilk -
2020 Poster: A Bayesian Perspective on Training Speed and Model Selection »
Clare Lyle · Lisa Schut · Robin Ru · Yarin Gal · Mark van der Wilk -
2020 Poster: Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty »
Miguel Monteiro · Loic Le Folgoc · Daniel Coelho de Castro · Nick Pawlowski · Bernardo Marques · Konstantinos Kamnitsas · Mark van der Wilk · Ben Glocker