Poster

Convergence of Actor-Critic with Multi-Layer Neural Networks

Haoxing Tian · Alex Olshevsky · Yannis Paschalidis

Great Hall & Hall B1+B2 (level 1) #1417
[ ]
Wed 13 Dec 8:45 a.m. PST — 10:45 a.m. PST

Abstract: The early theory of actor-critic methods considered convergence using linear function approximators for the policy and value functions. Recent work has established convergence using neural network approximators with a single hidden layer. In this work we are taking the natural next step and establish convergence using deep neural networks with an arbitrary number of hidden layers, thus closing a gap between theory and practice. We show that actor-critic updates projected on a ball around the initial condition will converge to a neighborhood where the average of the squared gradients is $\tilde{O} \left( 1/\sqrt{m} \right) + O \left( \epsilon \right)$, with $m$ being the width of the neural network and $\epsilon$ the approximation quality of the best critic neural network over the projected set.

Chat is not available.