Timezone: »
Deep model-based Reinforcement Learning (RL) has the potential to substantially improve the sample-efficiency of deep RL. While various challenges have long held it back, a number of papers have recently come out reporting success with deep model-based methods. This is a great development, but the lack of a consistent metric to evaluate such methods makes it difficult to compare various approaches. For example, the common single-task sample-efficiency metric conflates improvements due to model-based learning with various other aspects, such as representation learning, making it difficult to assess true progress on model-based RL. To address this, we introduce an experimental setup to evaluate model-based behavior of RL methods, inspired by work from neuroscience on detecting model-based behavior in humans and animals. Our metric based on this setup, the Local Change Adaptation (LoCA) regret, measures how quickly an RL method adapts to a local change in the environment. Our metric can identify model-based behavior, even if the method uses a poor representation and provides insight in how close a method's behavior is from optimal model-based behavior. We use our setup to evaluate the model-based behavior of MuZero on a variation of the classic Mountain Car task.
Author Information
Harm Van Seijen (Microsoft Research)
Hadi Nekoei (MILA)
Evan Racah (Mila, Université de Montréal)
Sarath Chandar (Mila / École Polytechnique de Montréal)
More from the Same Authors
-
2021 : IIRC: Incremental Implicitly-Refined Classification »
Mohamed Abdelsalam · Mojtaba Faramarzi · Shagun Sodhani · Sarath Chandar -
2022 : Agent-Controller Representations: Principled Offline RL with Rich Exogenous Information »
Riashat Islam · Manan Tomar · Alex Lamb · Hongyu Zang · Yonathan Efroni · Dipendra Misra · Aniket Didolkar · Xin Li · Harm Van Seijen · Remi Tachet des Combes · John Langford -
2022 : Replay Buffer With Local Forgetting for Adaptive Deep Model-Based Reinforcement Learning »
Ali Rahimi-Kalahroudi · Janarthanan Rajendran · Ida Momennejad · Harm Van Seijen · Sarath Chandar -
2019 Poster: Unsupervised State Representation Learning in Atari »
Ankesh Anand · Evan Racah · Sherjil Ozair · Yoshua Bengio · Marc-Alexandre Côté · R Devon Hjelm -
2019 Poster: Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning »
Harm Van Seijen · Mehdi Fatemi · Arash Tavakoli -
2019 Oral: Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning »
Harm Van Seijen · Mehdi Fatemi · Arash Tavakoli -
2017 Poster: ExtremeWeather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events »
Evan Racah · Christopher Beckham · Tegan Maharaj · Samira Ebrahimi Kahou · Mr. Prabhat · Chris Pal -
2017 Poster: Hybrid Reward Architecture for Reinforcement Learning »
Harm Van Seijen · Mehdi Fatemi · Romain Laroche · Joshua Romoff · Tavian Barnes · Jeffrey Tsang