Heart rate (HR) dynamics in response to workout intensity measure key aspects of an individual's fitness and cardiorespiratory health. Models of exercise physiology have been used to characterize cardiorespiratory fitness in well-controlled laboratory settings, but face additional challenges when applied to wearables in noisy, real-world settings. Here, we introduce a hybrid machine learning model that combines a physiological model of HR during exercise with complex neural networks in order to learn user-specific fitness representations. We apply this model at scale to a large set of workout data collected with wearables and show that it can accurately predict HR response to exercise demand in new workouts. We further show that the learned embeddings correlate with traditional metrics of cardiorespiratory fitness. Lastly, we illustrate how our model naturally incorporates and learn the effects of environmental factors such as temperature and humidity.