Timezone: »

 
Poster
Input-Output Equivalence of Unitary and Contractive RNNs
Melikasadat Emami · Mojtaba Sahraee Ardakan · Sundeep Rangan · Alyson Fletcher

Tue Dec 10 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #152
in Deep Learning -- Recurrent Networks »

Unitary recurrent neural networks (URNNs) have been proposed as a method to overcome the vanishing and exploding gradient problem in modeling data with long-term dependencies. A basic question is how restrictive is the unitary constraint on the possible input-output mappings of such a network? This works shows that for any contractive RNN with ReLU activations, there is a URNN with at most twice the number of hidden states and the identical input-output mapping. Hence, with ReLU activations, URNNs are as expressive as general RNNs. In contrast, for certain smooth activations, it is shown that the input-output mapping of an RNN cannot be matched with a URNN, even with an arbitrary number of states. The theoretical results are supported by experiments on modeling of slowly-varying dynamical systems.

Keywords: Recurrent Networks · Control Theory · Deep Learning · Applications -> Time Series Analysis; Theory
[ Paper]  [ Poster] 

Author Information

Melikasadat Emami (UCLA)
Mojtaba Sahraee Ardakan (UCLA)
Sundeep Rangan (NYU)
Alyson Fletcher (UCLA)

More from the Same Authors