Poster

Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation

Peihao Chen · Dongyu Ji · Kunyang Lin · Runhao Zeng · Thomas Li · Mingkui Tan · Chuang Gan

Hall J #702

Keywords: [ weakly-supervised learning ] [ Map Representation ] [ Vision-and-Language Navigation ]

[ Abstract ]
[ Poster [ OpenReview
Wed 30 Nov 2 p.m. PST — 4 p.m. PST
 
Spotlight presentation: Lightning Talks 4B-3
Wed 7 Dec 6 p.m. PST — 6:15 p.m. PST

Abstract:

We address a practical yet challenging problem of training robot agents to navigate in an environment following a path described by some language instructions. The instructions often contain descriptions of objects in the environment. To achieve accurate and efficient navigation, it is critical to build a map that accurately represents both spatial location and the semantic information of the environment objects. However, enabling a robot to build a map that well represents the environment is extremely challenging as the environment often involves diverse objects with various attributes. In this paper, we propose a multi-granularity map, which contains both object fine-grained details (\eg, color, texture) and semantic classes, to represent objects more comprehensively. Moreover, we propose a weakly-supervised auxiliary task, which requires the agent to localize instruction-relevant objects on the map. Through this task, the agent not only learns to localize the instruction-relevant objects for navigation but also is encouraged to learn a better map representation that reveals object information. We then feed the learned map and instruction to a waypoint predictor to determine the next navigation goal. Experimental results show our method outperforms the state-of-the-art by 4.0% and 4.6% w.r.t. success rate both in seen and unseen environments, respectively on VLN-CE dataset. The code is available at https://github.com/PeihaoChen/WS-MGMap.

Chat is not available.