Framing RNN as a kernel method: A neural ODE approach

Adeline Fermanian · Pierre Marion · Jean-Philippe Vert · Gérard Biau

Keywords: [ Theory ] [ Deep Learning ] [ Kernel Methods ]

[ Abstract ]
Thu 9 Dec 8:30 a.m. PST — 10 a.m. PST
Oral presentation: Oral Session 1: Deep Learning Theory and Causality
Tue 7 Dec midnight PST — 1 a.m. PST


Building on the interpretation of a recurrent neural network (RNN) as a continuous-time neural differential equation, we show, under appropriate conditions, that the solution of a RNN can be viewed as a linear function of a specific feature set of the input sequence, known as the signature. This connection allows us to frame a RNN as a kernel method in a suitable reproducing kernel Hilbert space. As a consequence, we obtain theoretical guarantees on generalization and stability for a large class of recurrent networks. Our results are illustrated on simulated datasets.

Chat is not available.