Parametric Computer-Aided Design (CAD) is the dominant paradigm for modernmechanical design. Training generative models to reason and generate parametricCAD can dramatically speed up design workflows. Pre-trained foundation modelshave shown great success in natural language processing and computer vision. The cross-domain knowledge embedded in these models holds significant potential for understanding geometry and performing complex reasoning about design. In this work, we develop generative models for CAD by leveraging pre-trained language models and apply them to manipulate engineering sketches. Our results demonstrate that models pre-trained on natural language can be finetuned on engineering sketches and achieve remarkable performance in various CAD generation scenarios.