In this paper we explore Bayesian approaches for the unsupervised estimation of probabilistic grammars, a family of distributions over discrete structures that includes hidden Markov models and probabilistic context-free grammars. We consider the use of Dirichlet priors, and we extend the correlated topic model framework to probabilistic grammars and derive a variational EM algorithm for efficient approximate inference. We experiment with the task of unsupervised grammar induction for natural language dependency parsing, and show that superior results can be achieved when using a logistic normal prior over probabilistic grammars.