Unsupervised Difficulty Estimation

"Evaluating difficulty and biases in machine learning models has become of ex- treme importance as current models are now being applied in real-world situations. In this paper we present a simple method for calculating a difficulty score based on the accumulation of losses for each sample during training. Our proposed method does not require any modification of the model neither any external supervision. We test and analyze our approach in two different settings that provide empirical evidence of the applicability of our method."