In this paper we study the problem of learning the structure of a discrete set of tokens from their interaction with other tokens. We focus on a setting where the tokens can be partitioned in a small number of classes with the property that all tokens from the same class interact in the same way with other tokens and we are interested in learning the class memberships from finite samples. We study this problem first from a complexity theoretic and an information theoretic viewpoint. Then we study the gradient flow dynamics for token embeddings and show that in certain settings this allows us to recover the clusters with small sample complexity.