Skip to yearly menu bar Skip to main content


Poster

The Reversible Residual Network: Backpropagation Without Storing Activations

Aidan Gomez · Mengye Ren · Raquel Urtasun · Roger Grosse

Pacific Ballroom #120

Keywords: [ Supervised Deep Networks ] [ Computer Vision ] [ Classification ] [ Efficient Training Methods ]


Abstract:

Residual Networks (ResNets) have demonstrated significant improvement over traditional Convolutional Neural Networks (CNNs) on image classification, increasing in performance as networks grow both deeper and wider. However, memory consumption becomes a bottleneck as one needs to store all the intermediate activations for calculating gradients using backpropagation. In this work, we present the Reversible Residual Network (RevNet), a variant of ResNets where each layer's activations can be reconstructed exactly from the next layer's. Therefore, the activations for most layers need not be stored in memory during backprop. We demonstrate the effectiveness of RevNets on CIFAR and ImageNet, establishing nearly identical performance to equally-sized ResNets, with activation storage requirements independent of depth.

Live content is unavailable. Log in and register to view live content