Rich Caruana, "Friends Don’t Let Friends Deploy Black-Box Models: The Importance of Intelligibility in Machine Learning"

In machine learning often a tradeoff must be made between accuracy and intelligibility: the most accurate models (deep nets, boosted trees and random forests) usually are not very intelligible, and the most intelligible models (logistic regression, small trees and decision lists) usually are less accurate. This tradeoff limits the accuracy of models that can be safely deployed in mission-critical applications such as healthcare where being able to understand, validate, edit, and ultimately trust a learned model is important. In this talk, I’ll present a case study where intelligibility is critical to uncover surprising patterns in the data that would have made deploying a black-box model risky. I’ll also show how distillation with intelligible models can be used to understand what is learned inside a black-box model such as a deep nets, and show a movie of what a deep net learns as it trains and then begins to overfit.