Timezone: »

 
Poster
Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation
Peihao Chen · Dongyu Ji · Kunyang Lin · Runhao Zeng · Thomas Li · Mingkui Tan · Chuang Gan

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #702
in Poster Session 4 »

We address a practical yet challenging problem of training robot agents to navigate in an environment following a path described by some language instructions. The instructions often contain descriptions of objects in the environment. To achieve accurate and efficient navigation, it is critical to build a map that accurately represents both spatial location and the semantic information of the environment objects. However, enabling a robot to build a map that well represents the environment is extremely challenging as the environment often involves diverse objects with various attributes. In this paper, we propose a multi-granularity map, which contains both object fine-grained details (\eg, color, texture) and semantic classes, to represent objects more comprehensively. Moreover, we propose a weakly-supervised auxiliary task, which requires the agent to localize instruction-relevant objects on the map. Through this task, the agent not only learns to localize the instruction-relevant objects for navigation but also is encouraged to learn a better map representation that reveals object information. We then feed the learned map and instruction to a waypoint predictor to determine the next navigation goal. Experimental results show our method outperforms the state-of-the-art by 4.0% and 4.6% w.r.t. success rate both in seen and unseen environments, respectively on VLN-CE dataset. The code is available at https://github.com/PeihaoChen/WS-MGMap.

Keywords: weakly-supervised learning · Map Representation · Vision-and-Language Navigation
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Peihao Chen (South China University of Technology)
Dongyu Ji (South China University of Technology)
Kunyang Lin (South China University of Technology)
Runhao Zeng (South China University of Technology)
Thomas Li (AIIT, Peking University)
Mingkui Tan (South China University of Technology)
Chuang Gan (UMass Amherst/ MIT-IBM Watson AI Lab)

More from the Same Authors