Hierarchical topic models (HTMs)---especially those based on Bayesian deep learning---are gaining increasing attention from the ML community. However, in contrast to their flat counterparts, their proper evaluation is rarely addressed. We propose several measures to evaluate HTMs in terms of their (branch-wise and layer-wise) topic hierarchy. We apply these measures to benchmark several HTMs on a wide range of datasets. We compare neural HTMs to traditional statistical HTMs in topic quality and interpretability. Our findings may help better judge advantages and disadvantages in different deep hierarchical topic models and drive future research in this area.