Timezone: »
Reinforcement learning (RL) in partially observable settings is challenging because the agent’s observations are not Markov. Recently proposed methods can learn variable-order Markov models of the underlying process but have steep memory requirements and are sensitive to aliasing between observation histories due to sensor noise. This paper proposes dynamic-depth context tree weighting (D2-CTW), a model-learning method that addresses these limitations. D2-CTW dynamically expands a suffix tree while ensuring that the size of the model, but not its depth, remains bounded. We show that D2-CTW approximately matches the performance of state-of-the-art alternatives at stochastic time-series prediction while using at least an order of magnitude less memory. We also apply D2-CTW to model-based RL, showing that, on tasks that require memory of past observations, D2-CTW can learn without prior knowledge of a good state representation, or even the length of history upon which such a representation should depend.
Author Information
Joao V Messias (Morpheus Labs)
Shimon Whiteson (Oxford University)
More from the Same Authors
-
2023 : Policy-Guided Diffusion »
Matthew T Jackson · Michael Matthews · Cong Lu · Jakob Foerster · Shimon Whiteson -
2023 : Discovering Temporally-Aware Reinforcement Learning Algorithms »
Matthew T Jackson · Chris Lu · Louis Kirsch · Robert Lange · Shimon Whiteson · Jakob Foerster -
2023 : JaxMARL: Multi-Agent RL Environments in JAX »
Alexander Rutherford · Benjamin Ellis · Matteo Gallici · Jonathan Cook · Andrei Lupu · Garðar Ingvarsson · Timon Willi · Akbir Khan · Christian Schroeder de Witt · Alexandra Souly · Saptarashmi Bandyopadhyay · Mikayel Samvelyan · Minqi Jiang · Robert Lange · Shimon Whiteson · Bruno Lacerda · Nick Hawes · Tim Rocktäschel · Chris Lu · Jakob Foerster -
2023 Poster: SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning »
Benjamin Ellis · Jonathan Cook · Skander Moalla · Mikayel Samvelyan · Mingfei Sun · Anuj Mahajan · Jakob Foerster · Shimon Whiteson -
2023 Poster: The Waymo Open Sim Agents Challenge »
Nico Montali · John Lambert · Paul Mougin · Alex Kuefler · Nicholas Rhinehart · Michelle Li · Cole Gulino · Tristan Emrich · Zoey Yang · Shimon Whiteson · Brandyn White · Dragomir Anguelov -
2023 Poster: Recurrent Hypernetworks are Surprisingly Strong in Meta-RL »
Jacob Beck · Risto Vuorio · Zheng Xiong · Shimon Whiteson -
2023 Poster: Discovering General Reinforcement Learning Algorithms with Adversarial Environment Design »
Matthew T Jackson · Minqi Jiang · Jack Parker-Holder · Risto Vuorio · Chris Lu · Greg Farquhar · Shimon Whiteson · Jakob Foerster -
2022 Poster: In Defense of the Unitary Scalarization for Deep Multi-Task Learning »
Vitaly Kurin · Alessandro De Palma · Ilya Kostrikov · Shimon Whiteson · Pawan K Mudigonda -
2022 Poster: Equivariant Networks for Zero-Shot Coordination »
Darius Muglich · Christian Schroeder de Witt · Elise van der Pol · Shimon Whiteson · Jakob Foerster -
2019 : Bayes-Adaptive Deep Reinforcement Learning via Meta-Learning - Invited Talk »
Shimon Whiteson