In this paper, we propose RetroLaDi, a 3D-aware face image generation method by Retrofitting a 2D Latent Diffusion model. The core procedure is to integrate and reconcile external 3D priors from 3DMMs with the internal knowledge in a pretrained 2D diffusion model.Experimental results demonstrate its ability to generate high-fidelity face images with precise controllability than state-of-the-art 2D-based and 3D-based controllable face synthesis methods.