Timezone: »
The task of vision-and-language navigation (VLN) requires an agent to follow text instructions to find its way through simulated household environments. A prominent challenge is to train an agent capable of generalising to new environments at test time, rather than one that simply memorises trajectories and visual details observed during training. We propose a new learning strategy that learns both from observations and generated counterfactual environments. We describe an effective algorithm to generate counterfactual observations on the fly for VLN, as linear combinations of existing environments. Simultaneously, we encourage the agent's actions to remain stable between original and counterfactual environments through our novel training objective-effectively removing the spurious features that otherwise bias the agent. Our experiments show that this technique provides significant improvements in generalisation on benchmarks for Room-to-Room navigation and Embodied Question Answering.
Author Information
Amin Parvaneh (University of Adelaide)
Ehsan Abbasnejad (University of Adelaide)
Damien Teney (University of Adelaide)
Javen Qinfeng Shi (University of Adelaide)
Anton van den Hengel (University of Adelaide)
Related Events (a corresponding poster, oral, or spotlight)
-
2020 Poster: Counterfactual Vision-and-Language Navigation: Unravelling the Unseen »
Thu. Dec 10th 05:00 -- 07:00 AM Room Poster Session 4 #1232
More from the Same Authors
-
2022 : Distributionally Robust Bayesian Optimization with φ-divergences »
Hisham Husain · Vu Nguyen · Anton van den Hengel -
2023 Poster: Distributionally Robust Bayesian Optimization with $\varphi$-divergences »
Hisham Husain · Vu Nguyen · Anton van den Hengel -
2023 Poster: RanPAC: Random Projections and Pre-trained Models for Continual Learning »
Mark McDonnell · Dong Gong · Amin Parvaneh · Ehsan Abbasnejad · Anton van den Hengel -
2023 Poster: ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets »
Damien Teney · Yong Lin · Seong Joon Oh · Ehsan Abbasnejad -
2022 Poster: Truncated Matrix Power Iteration for Differentiable DAG Learning »
Zhen Zhang · Ignavier Ng · Dong Gong · Yuhang Liu · Ehsan Abbasnejad · Mingming Gong · Kun Zhang · Javen Qinfeng Shi -
2020 Poster: On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law »
Damien Teney · Ehsan Abbasnejad · Kushal Kafle · Robik Shrestha · Christopher Kanan · Anton van den Hengel -
2015 Poster: Deeply Learning the Messages in Message Passing Inference »
Guosheng Lin · Chunhua Shen · Ian Reid · Anton van den Hengel -
2014 Poster: Encoding High Dimensional Local Features by Sparse Coding Based Fisher Vectors »
Lingqiao Liu · Chunhua Shen · Lei Wang · Anton van den Hengel · Chao Wang -
2009 Poster: Positive Semidefinite Metric Learning with Boosting »
Chunhua Shen · Junae Kim · Lei Wang · Anton van den Hengel