Reverse-engineering recurrent networks performing Gaussian process regression

When performing tasks in noisy environments, biological and artificial agents often approximate optimal Bayesian inference. However, Bayes-like behavior alone does not determine whether a system maintains an internal representation of its uncertainty or merely relies on heuristics; rather, this determination requires directly probing and reverse-engineering the system's internal representation, which is more tractable in artificial than in biological networks. Thus, we investigate neural representations of uncertainty in gated recurrent neural networks trained to perform sequential Gaussian process (GP) regression, a setting where the target Bayesian computation is exactly specified, uncertainty is explicit, and an expanding dataset must be compressed into a fixed recurrent state. We show that GRUs solve this task accurately and that they do so by learning to represent the subspace spanned by the top eigenfunctions of the true kernel and performing recursive Bayesian regression in this truncated basis. We find that the GRU's update gate supports approximate expectation-parameter updates, and show that we can not only decode the expectation parameters from the hidden state but also causally perturb that decoded subspace to alter predictions coherently. Our results show how finite-state recurrent networks can implement sequential probabilistic inference, and offer a methodology for identifying belief-state representations in both artificial and biological dynamical systems.