Unsupervised Learning of Equivariant Structure from Sequences

Takeru Miyato · Masanori Koyama · Kenji Fukumizu

Hall J #228

Keywords: [ Disentanglement ] [ Equivariance ] [ Unsupervised Learning ] [ Symmetry ]

[ Abstract ]
[ Paper [ Poster [ OpenReview
Thu 1 Dec 9 a.m. PST — 11 a.m. PST


In this study, we present \textit{meta-sequential prediction} (MSP), an unsupervised framework to learn the symmetry from the time sequence of length at least three. Our method leverages the stationary property~(e.g. constant velocity, constant acceleration) of the time sequence to learn the underlying equivariant structure of the dataset by simply training the encoder-decoder model to be able to predict the future observations. We will demonstrate that, with our framework, the hidden disentangled structure of the dataset naturally emerges as a by-product by applying \textit{simultaneous block-diagonalization} to the transition operators in the latent space, the procedure which is commonly used in representation theory to decompose the feature-space based on the type of response to group actions.We will showcase our method from both empirical and theoretical perspectives.Our result suggests that finding a simple structured relation and learning a model with extrapolation capability are two sides of the same coin. The code is available at

Chat is not available.