Spotlight Poster

Double Gumbel Q-Learning

David Yu-Tung Hui · Aaron Courville · Pierre-Luc Bacon

Great Hall & Hall B1+B2 (level 1) #1502
[ ] [ Project Page ]
Wed 13 Dec 3 p.m. PST — 5 p.m. PST

Abstract:

We show that Deep Neural Networks introduce two heteroscedastic Gumbel noise sources into Q-Learning. To account for these noise sources, we propose Double Gumbel Q-Learning, a Deep Q-Learning algorithm applicable for both discrete and continuous control. In discrete control, we derive a closed-form expression for the loss function of our algorithm. In continuous control, this loss function is intractable and we therefore derive an approximation with a hyperparameter whose value regulates pessimism in Q-Learning. We present a default value for our pessimism hyperparameter that enables DoubleGum to outperform DDPG, TD3, SAC, XQL, quantile regression, and Mixture-of-Gaussian Critics in aggregate over 33 tasks from DeepMind Control, MuJoCo, MetaWorld, and Box2D and show that tuning this hyperparameter may further improve sample efficiency.

Chat is not available.