Skip to yearly menu bar Skip to main content


Poster

Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization

Hyeonwoo Noh · Tackgeun You · Jonghwan Mun · Bohyung Han

Pacific Ballroom #143

Keywords: [ Deep Learning ] [ Supervised Deep Networks ] [ Computer Vision ]


Abstract:

Overfitting is one of the most critical challenges in deep neural networks, and there are various types of regularization methods to improve generalization performance. Injecting noises to hidden units during training, e.g., dropout, is known as a successful regularizer, but it is still not clear enough why such training techniques work well in practice and how we can maximize their benefit in the presence of two conflicting objectives---optimizing to true data distribution and preventing overfitting by regularization. This paper addresses the above issues by 1) interpreting that the conventional training methods with regularization by noise injection optimize the lower bound of the true objective and 2) proposing a technique to achieve a tighter lower bound using multiple noise samples per training example in a stochastic gradient descent iteration. We demonstrate the effectiveness of our idea in several computer vision applications.

Live content is unavailable. Log in and register to view live content