Timezone: »
A residual network may be regarded as a discretization of an ordinary differential equation (ODE) which, in the limit of time discretization, defines a continuous-depth network. Although important steps have been taken to realize the advantages of such continuous formulations, most current techniques assume identical layers. Indeed, existing works throw into relief the myriad difficulties of learning an infinite-dimensional parameter in a continuous-depth neural network. To this end, we introduce a shooting formulation which shifts the perspective from parameterizing a network layer-by-layer to parameterizing over optimal networks described only by a set of initial conditions. For scalability, we propose a novel particle-ensemble parameterization which fully specifies the optimal weight trajectory of the continuous-depth neural network. Our experiments show that our particle-ensemble shooting formulation can achieve competitive performance. Finally, though the current work is inspired by continuous-depth neural networks, the particle-ensemble shooting formulation also applies to discrete-time networks and may lead to a new fertile area of research in deep learning parameterization.
Author Information
François-Xavier Vialard (University Gustave Eiffel)
Roland Kwitt (University of Salzburg)
Susan Wei (University of Melbourne)
Marc Niethammer (UNC Chapel Hill)
Related Events (a corresponding poster, oral, or spotlight)
-
2020 Poster: A shooting formulation of deep learning »
Thu. Dec 10th 05:00 -- 07:00 PM Room Poster Session 5 #1408
More from the Same Authors
-
2022 Poster: Compositional Generalization in Unsupervised Compositional Representation Learning: A Study on Disentanglement and Emergent Language »
Zhenlin Xu · Marc Niethammer · Colin Raffel -
2022 Poster: On Measuring Excess Capacity in Neural Networks »
Florian Graf · Sebastian Zeng · Bastian Rieck · Marc Niethammer · Roland Kwitt -
2021 Poster: Accurate Point Cloud Registration with Robust Optimal Transport »
Zhengyang Shen · Jean Feydy · Peirong Liu · Ariel H Curiale · Ruben San Jose Estepar · Raul San Jose Estepar · Marc Niethammer -
2020 : Closing Remarks »
Frederic Chazal · Smita Krishnaswamy · Roland Kwitt · Karthikeyan Natesan Ramamurthy · Bastian Rieck · Yuhei Umeda · Guy Wolf -
2020 Workshop: Topological Data Analysis and Beyond »
Bastian Rieck · Frederic Chazal · Smita Krishnaswamy · Roland Kwitt · Karthikeyan Natesan Ramamurthy · Yuhei Umeda · Guy Wolf -
2020 : Opening Remarks »
Frederic Chazal · Smita Krishnaswamy · Roland Kwitt · Karthikeyan Natesan Ramamurthy · Bastian Rieck · Yuhei Umeda · Guy Wolf -
2020 Poster: Faster Wasserstein Distance Estimation with the Sinkhorn Divergence »
Lénaïc Chizat · Pierre Roussillon · Flavien Léger · François-Xavier Vialard · Gabriel Peyré -
2019 Poster: Region-specific Diffeomorphic Metric Mapping »
Zhengyang Shen · Francois-Xavier Vialard · Marc Niethammer -
2017 Poster: Deep Learning with Topological Signatures »
Christoph Hofer · Roland Kwitt · Marc Niethammer · Andreas Uhl -
2015 Poster: Statistical Topological Data Analysis - A Kernel Perspective »
Roland Kwitt · Stefan Huber · Marc Niethammer · Weili Lin · Ulrich Bauer