Timezone: »

Zap Q-Learning With Nonlinear Function Approximation
Shuhang Chen · Adithya M Devraj · Fan Lu · Ana Busic · Sean Meyn

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #549

Zap Q-learning is a recent class of reinforcement learning algorithms, motivated primarily as a means to accelerate convergence. Stability theory has been absent outside of two restrictive classes: the tabular setting, and optimal stopping. This paper introduces a new framework for analysis of a more general class of recursive algorithms known as stochastic approximation. Based on this general theory, it is shown that Zap Q-learning is consistent under a non-degeneracy assumption, even when the function approximation architecture is nonlinear. Zap Q-learning with neural network function approximation emerges as a special case, and is tested on examples from OpenAI Gym. Based on multiple experiments with a range of neural network sizes, it is found that the new algorithms converge quickly and are robust to choice of function approximation architecture.

Author Information

Shuhang Chen (University of Florida)
Adithya M Devraj (Stanford University)
Fan Lu (University of Florida)
Ana Busic (INRIA)
Sean Meyn (University of Florida)

More from the Same Authors