We explore the relationship between neural network architectures and model accuracy under differential privacy constraints. Our findings show that architectures that perform well without differential privacy, do not necessarily do so with differential privacy. This shows that extant knowledge on neural network architecture design cannot be seamlessly translated into the differential privacy context. Moreover, as neural architecture search consumes privacy budget, future research is required to better understand the relationship between neural network architectures and model accuracy to enable better architecture design choices under differential privacy constraints.