Timezone: »

 
Poster
Provably Efficient Causal Reinforcement Learning with Confounded Observational Data
Lingxiao Wang · Zhuoran Yang · Zhaoran Wang

Tue Dec 07 08:30 AM -- 10:00 AM (PST) @
in Poster Session 1 »

Empowered by neural networks, deep reinforcement learning (DRL) achieves tremendous empirical success. However, DRL requires a large dataset by interacting with the environment, which is unrealistic in critical scenarios such as autonomous driving and personalized medicine. In this paper, we study how to incorporate the dataset collected in the offline setting to improve the sample efficiency in the online setting. To incorporate the observational data, we face two challenges. (a) The behavior policy that generates the observational data may depend on unobserved random variables (confounders), which affect the received rewards and transition dynamics. (b) Exploration in the online setting requires quantifying the uncertainty given both the observational and interventional data. To tackle such challenges, we propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner. DOVI explicitly adjusts for the confounding bias in the observational data, where the confounders are partially observed or unobserved. In both cases, such adjustments allow us to construct the bonus based on a notion of information gain, which takes into account the amount of information acquired from the offline setting. In particular, we prove that the regret of DOVI is smaller than the optimal regret achievable in the pure online setting when the confounded observational data are informative upon the adjustments.

Keywords: Deep Learning · Reinforcement Learning and Planning · Causality

Author Information

Lingxiao Wang (Northwestern University)
Zhuoran Yang (Princeton)
Zhaoran Wang (Princeton University)

More from the Same Authors