`

Timezone: »

 
Poster
Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
Karl Pertsch · Oleh Rybkin · Frederik Ebert · Shenghao Zhou · Dinesh Jayaraman · Chelsea Finn · Sergey Levine

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #562
in Poster Session 2 »

The ability to predict and plan into the future is fundamental for agents acting in the world. To reach a faraway goal, we predict trajectories at multiple timescales, first devising a coarse plan towards the goal and then gradually filling in details. In contrast, current learning approaches for visual prediction and planning fail on long-horizon tasks as they generate predictions (1)~without considering goal information, and (2)~at the finest temporal resolution, one step at a time. In this work we propose a framework for visual prediction and planning that is able to overcome both of these limitations. First, we formulate the problem of predicting towards a goal and propose the corresponding class of latent space goal-conditioned predictors (GCPs). GCPs significantly improve planning efficiency by constraining the search space to only those trajectories that reach the goal. Further, we show how GCPs can be naturally formulated as hierarchical models that, given two observations, predict an observation between them, and by recursively subdividing each part of the trajectory generate complete sequences. This divide-and-conquer strategy is effective at long-term prediction, and enables us to design an effective hierarchical planning algorithm that optimizes trajectories in a coarse-to-fine manner. We show that by using both goal-conditioning and hierarchical prediction, GCPs enable us to solve visual planning tasks with much longer horizon than previously possible. See prediction and planning videos on the supplementary website: sites.google.com/view/video-gcp.

Keywords: Applications -> Robotics; Reinforcement Learning and Planning -> Exploration; Reinforcement Learning and Planning -> Reinforcem · Algorithms -> Multitask and Transfer Learning

Author Information

Karl Pertsch (University of Southern California)
Oleh Rybkin (University of Pennsylvania)

I am a Ph.D. student in the GRASP laboratory at the University of Pennsylvania, where I work on computer vision and deep learning with Kostas Daniilidis. Previously, I received my bachelor's degree from Czech Technical University in Prague, where I was advised by Tomas Pajdla. I have spent two summers at INRIA and TiTech, with Josef Sivic and Akihiko Torii respectively. I am working in artificial intelligence, computer vision, and robotics. More specifically, my main interest is machine understanding of intuitive physics for real-world robotic manipulation. My latest work has been on motion understanding via video prediction. During my bachelor's, I also worked on camera geometry for structure from motion.

Frederik Ebert (UC Berkeley)
Shenghao Zhou (University of Pennsylvania)
Dinesh Jayaraman (University of Pennsylvania)

I am an assistant professor at UPenn’s GRASP lab. I lead the Perception, Action, and Learning (PAL) Research Group, where we work on problems at the intersection of computer vision, machine learning, and robotics.

Chelsea Finn (Stanford)
Sergey Levine (UC Berkeley)

More from the Same Authors