Timezone: »

 
Poster
Propagating Uncertainty in Reinforcement Learning via Wasserstein Barycenters
Alberto Maria Metelli · Amarildo Likmeta · Marcello Restelli

Tue Dec 10 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #194
in Reinforcement Learning and Planning -- Exploration »

How does the uncertainty of the value function propagate when performing temporal difference learning? In this paper, we address this question by proposing a Bayesian framework in which we employ approximate posterior distributions to model the uncertainty of the value function and Wasserstein barycenters to propagate it across state-action pairs. Leveraging on these tools, we present an algorithm, Wasserstein Q-Learning (WQL), starting in the tabular case and then, we show how it can be extended to deal with continuous domains. Furthermore, we prove that, under mild assumptions, a slight variation of WQL enjoys desirable theoretical properties in the tabular setting. Finally, we present an experimental campaign to show the effectiveness of WQL on finite problems, compared to several RL algorithms, some of which are specifically designed for exploration, along with some preliminary results on Atari games.

Keywords: Reinforcement Learning and Planning ·

Author Information

Alberto Maria Metelli (Politecnico di Milano)
Amarildo Likmeta (Politecnico di Milano)
Marcello Restelli (Politecnico di Milano)

More from the Same Authors