Sky sphere representation in language models

We analyze whether language models of size ∼100B have a representation of the night sky map that is decodable from their residual stream. We find that most of the considered open-source models do have such a representation, and it often even surfaces to the top principal components on prompts that ask questions like “what is close to this object in the night sky”. In most models this representation showed significant scores in LOO testing, containing up to 80–90% of variance (R2-score) and having median angular error down to 10◦–20◦. We verify that our representation is not a simple leak from a correlated flat representation. To our knowledge, this representation is the first example of a curved high-dimensional irreducible feature manifold.