Timezone: »

 
Poster
Strictly Batch Imitation Learning by Energy-based Distribution Matching
Daniel Jarrett · Ioana Bica · Mihaela van der Schaar

Thu Dec 10 09:00 AM -- 11:00 AM (PST) @ Poster Session 5 #1384
in Poster Session 6 »

Consider learning a policy purely on the basis of demonstrated behavior---that is, with no access to reinforcement signals, no knowledge of transition dynamics, and no further interaction with the environment. This strictly batch imitation learning problem arises wherever live experimentation is costly, such as in healthcare. One solution is simply to retrofit existing algorithms for apprenticeship learning to work in the offline setting. But such an approach leans heavily on off-policy evaluation or offline model estimation, and can be indirect and inefficient. We argue that a good solution should be able to explicitly parameterize a policy (i.e. respecting action conditionals), implicitly learn from rollout dynamics (i.e. leveraging state marginals), and---crucially---operate in an entirely offline fashion. To address this challenge, we propose a novel technique by energy-based distribution matching (EDM): By identifying parameterizations of the (discriminative) model of a policy with the (generative) energy function for state distributions, EDM yields a simple but effective solution that equivalently minimizes a divergence between the occupancy measure for the demonstrator and a model thereof for the imitator. Through experiments with application to control and healthcare settings, we illustrate consistent performance gains over existing algorithms for strictly batch imitation learning.

Author Information

Daniel Jarrett (University of Cambridge)
Ioana Bica (University of Oxford)
Mihaela van der Schaar (University of Cambridge)

More from the Same Authors