Timezone: »

 
Oral
A shooting formulation of deep learning
François-Xavier Vialard · Roland Kwitt · Susan Wei · Marc Niethammer

Thu Dec 10 06:00 AM -- 06:15 AM (PST) @ Orals & Spotlights: Deep Learning
in Orals & Spotlights Track 28: Deep Learning »

A residual network may be regarded as a discretization of an ordinary differential equation (ODE) which, in the limit of time discretization, defines a continuous-depth network. Although important steps have been taken to realize the advantages of such continuous formulations, most current techniques assume identical layers. Indeed, existing works throw into relief the myriad difficulties of learning an infinite-dimensional parameter in a continuous-depth neural network. To this end, we introduce a shooting formulation which shifts the perspective from parameterizing a network layer-by-layer to parameterizing over optimal networks described only by a set of initial conditions. For scalability, we propose a novel particle-ensemble parameterization which fully specifies the optimal weight trajectory of the continuous-depth neural network. Our experiments show that our particle-ensemble shooting formulation can achieve competitive performance. Finally, though the current work is inspired by continuous-depth neural networks, the particle-ensemble shooting formulation also applies to discrete-time networks and may lead to a new fertile area of research in deep learning parameterization.

Author Information

François-Xavier Vialard (University Gustave Eiffel)
Roland Kwitt (University of Salzburg)
Susan Wei (University of Melbourne)
Marc Niethammer (UNC Chapel Hill)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors