Timezone: »

 
Poster
Dynamic-Depth Context Tree Weighting
Joao V Messias · Shimon Whiteson

Wed Dec 06 06:30 PM -- 10:30 PM (PST) @ Pacific Ballroom #201
in Posters Wed »

Reinforcement learning (RL) in partially observable settings is challenging because the agent’s observations are not Markov. Recently proposed methods can learn variable-order Markov models of the underlying process but have steep memory requirements and are sensitive to aliasing between observation histories due to sensor noise. This paper proposes dynamic-depth context tree weighting (D2-CTW), a model-learning method that addresses these limitations. D2-CTW dynamically expands a suffix tree while ensuring that the size of the model, but not its depth, remains bounded. We show that D2-CTW approximately matches the performance of state-of-the-art alternatives at stochastic time-series prediction while using at least an order of magnitude less memory. We also apply D2-CTW to model-based RL, showing that, on tasks that require memory of past observations, D2-CTW can learn without prior knowledge of a good state representation, or even the length of history upon which such a representation should depend.

Keywords: Online Learning · Reinforcement Learning · Model-Based RL
[ Code]  [ Paper] 

Author Information

Joao V Messias (Morpheus Labs)
Shimon Whiteson (Oxford University)

More from the Same Authors