Estimating Uncertainty in Safety-Critical Deep Learning Models

This paper creates a regression model using the”Deep Ensemble” technique for predicting the Remaining UsefulLife (RUL) of an aircraft engine, which is a safety-criticalapplication. The run-to-failure turbo engine degradation datasethas been used, which is widely considered as a benchmarkdataset for aero engine predictive maintenance work.This paper identifies a gap in the previous works and provides asolution to that. The previous works only focused on developingpredictive models using classical Neural Network architecturesfor estimating the RUL of the turbo engines, however those worksdid not estimate the uncertainty from the predictive models.Since it is critical to know how certain the model is about itsown prediction, hence this project addresses that shortfall. Asany classical Neural Network architectures do not provide anymechanism to obtain the uncertainty from the model, hence aprobabilistic approach has been taken to modify the existingNeural Network architecture to obtain uncertainty estimates.The probabilistic method ”Deep Ensemble” is used and NegativeLog Likelihood (NLL) is applied as a training criterion forthe model. This model is run on simulated data-set from fourdifferent fleets of aero engine and the model’s prediction error isevaluated using Root Mean Square Error (RMSE) and Coefficientof Determination (R2). Various experiments showed that the model is onlyconfident when the prediction error is low, on the other handwhen the error rate is high, the uncertainty estimate is also high,which means that the model is aware of its own uncertainty. Theerror rate for two out of the four sets of fleet data have higherprediction error likewise with the uncertainty values, hence itprovides an useful insight on the confidence of the model, whichis critical for decision making in these types of safety-criticalapplications.