Timezone: »

Initialization of ReLUs for Dynamical Isometry
Rebekka Burkholz · Alina Dubatovka

Thu Dec 12 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #160

Deep learning relies on good initialization schemes and hyperparameter choices prior to training a neural network. Random weight initializations induce random network ensembles, which give rise to the trainability, training speed, and sometimes also generalization ability of an instance. In addition, such ensembles provide theoretical insights into the space of candidate models of which one is selected during training. The results obtained so far rely on mean field approximations that assume infinite layer width and that study average squared signals. We derive the joint signal output distribution exactly, without mean field assumptions, for fully-connected networks with Gaussian weights and biases, and analyze deviations from the mean field results. For rectified linear units, we further discuss limitations of the standard initialization scheme, such as its lack of dynamical isometry, and propose a simple alternative that overcomes these by initial parameter sharing.

Author Information

Rebekka Burkholz (Harvard University)
Alina Dubatovka (ETH Zurich)

More from the Same Authors

  • 2021 Poster: Scaling up Continuous-Time Markov Chains Helps Resolve Underspecification »
    Alkis Gotovos · Rebekka Burkholz · John Quackenbush · Stefanie Jegelka
  • 2019 : Coffee/Poster session 1 »
    Shiro Takagi · Khurram Javed · Johanna Sommer · Amr Sharaf · Pierluca D'Oro · Ying Wei · Sivan Doveh · Colin White · Santiago Gonzalez · Cuong Nguyen · Mao Li · Tianhe Yu · Tiago Ramalho · Masahiro Nomura · Ahsan Alvi · Jean-Francois Ton · W. Ronny Huang · Jessica Lee · Sebastian Flennerhag · Michael Zhang · Abram Friesen · Paul Blomstedt · Alina Dubatovka · Sergey Bartunov · Subin Yi · Iaroslav Shcherbatyi · Christian Simon · Zeyuan Shang · David MacLeod · Lu Liu · Liam Fowl · Diego Mesquita · Deirdre Quillen