Timezone: »

 
Poster
Bridging the Imitation Gap by Adaptive Insubordination
Luca Weihs · Unnat Jain · Iou-Jen Liu · Jordi Salvador · Svetlana Lazebnik · Aniruddha Kembhavi · Alex Schwing

Fri Dec 10 08:30 AM -- 10:00 AM (PST) @
in Poster Session 8 »

In practice, imitation learning is preferred over pure reinforcement learning whenever it is possible to design a teaching agent to provide expert supervision. However, we show that when the teaching agent makes decisions with access to privileged information that is unavailable to the student, this information is marginalized during imitation learning, resulting in an "imitation gap" and, potentially, poor results. Prior work bridges this gap via a progression from imitation learning to reinforcement learning. While often successful, gradual progression fails for tasks that require frequent switches between exploration and memorization. To better address these tasks and alleviate the imitation gap we propose 'Adaptive Insubordination' (ADVISOR). ADVISOR dynamically weights imitation and reward-based reinforcement learning losses during training, enabling on-the-fly switching between imitation and exploration. On a suite of challenging tasks set within gridworlds, multi-agent particle environments, and high-fidelity 3D simulators, we show that on-the-fly switching with ADVISOR outperforms pure imitation, pure reinforcement learning, as well as their sequential and parallel combinations.

Keywords: Reinforcement Learning and Planning

Author Information

Luca Weihs (Allen Institute for Artificial Intelligence)
Unnat Jain (University of Illinois at Urbana-Champaign (UIUC))
Iou-Jen Liu (University of Illinois at Urbana-Champaign)
Jordi Salvador (Allen Institute for AI)
Svetlana Lazebnik (UIUC)
Aniruddha Kembhavi (Allen Institute for Artificial Intelligence (AI2))
Alex Schwing (University of Illinois at Urbana-Champaign)

More from the Same Authors