Memory-based Self-Explainable Early Exit Networks

Early Exit Neural Networks are networks that accelerate inference by allowing samples to exit through internal classifiers. One of the understudied areas related to these networks is whether early exits can provide interpretability alongside the prediction. In this work, we propose a memory-based self-interpretable design to make the internal classifiers more interpretable. Our design uses a sparse content-based attention mechanism to select relevant examples from a memory set and uses them to aid and explain the decision process. We show that, compared to alternative self-explainable designs, our design better preserves the performance of black-box baselines while providing means to inspect its inner mechanisms.