Phase transitions and structure formation in learning local rules

We study a teacher-student rule learning scenario, where the teacher is determined by a local rule and the student model is a uniform tensor-network attention model. The student model also implements a map from variable-size binary inputs to the latent space $\mathcal{V}=\mathds{R}^d$, where $d$ is the bond dimension of the student model. Using gradient descent learning we find a second-order phase transition in the test error. At the transition we observe a sudden drop in the effective dimension of the mapped training data. We also find that small-effective dimension corresponds to structure formation in the latent space $\mathcal{V}$.