Symmetries

Modern neural networks are powerful function approximators which manipulate complex input data to extract information useful for one or more tasks. Extracting information is a process which transforms data to make it simpler and simpler to use. For example, binary classification might start from high dimensional images, and progressively simplify this data to shrink it down to a single number between 0 and 1: the probability that the original image contained a dog. Many transformations of the input image should lead to the same output. For example, changing background colour, rotating images, and so on, should not affect the answer provided by the neural network. When the transformations applied to the input are invertible, we are dealing with symmetries of the inputs. And a question arises of whether knowledge of such symmetries can help researchers devise better neural networks. In this section, we will visit the subject of symmetries, their benefits, and see examples of their usage.