Poster
Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
Haw-Shiuan Chang · Erik Learned-Miller · Andrew McCallum

Tue Dec 5th 06:30 -- 10:30 PM @ Pacific Ballroom #228 #None
in Posters Tue »

Self-paced learning and hard example mining re-weight training instances to improve learning accuracy. This paper presents two improved alternatives based on lightweight estimates of sample uncertainty in stochastic gradient descent (SGD): the variance in predicted probability of the correct class across iterations of mini-batch SGD, and the proximity of the correct class probability to the decision threshold. Extensive experimental results on six datasets show that our methods reliably improve accuracy in various network architectures, including additional gains on top of other popular training techniques, such as residual learning, momentum, ADAM, batch normalization, dropout, and distillation.

Keywords: Active Learning · Supervised Deep Networks · Computer Vision · Classification · Natural Language Processing · Optimization for Deep Networks · Efficient Training Methods

Author Information

Haw-Shiuan Chang (UMass, Amherst)
Erik Learned-Miller (UMass Amherst)
Andrew McCallum (UMass Amherst)

More from the Same Authors