This paper focuses on estimating probability distributions over the set of 3D ro- tations (SO(3)) using deep neural networks. Learning to regress models to the set of rotations is inherently difficult due to differences in topology between R^N and SO(3). We overcome this issue by using a neural network to out- put the parameters for a matrix Fisher distribution since these parameters are homeomorphic to R^9 . By using a negative log likelihood loss for this distri- bution we get a loss which is convex with respect to the network outputs. By optimizing this loss we improve state-of-the-art on several challenging applica- ble datasets, namely Pascal3D+, ModelNet10-SO(3). Our code is available at https://github.com/Davmo049/Publicproborientationestimationwithmatrix _fisherdistributions