Skip to yearly menu bar Skip to main content


Poster

Teaching Machines to Describe Images with Natural Language Feedback

Huan Ling · Sanja Fidler

Pacific Ballroom #87

Keywords: [ Computer Vision ] [ Visual Question Answering ]


Abstract:

Robots will eventually be part of every household. It is thus critical to enable algorithms to learn from and be guided by non-expert users. In this paper, we bring a human in the loop, and enable a human teacher to give feedback to a learning agent in the form of natural language. A descriptive sentence can provide a stronger learning signal than a numeric reward in that it can easily point to where the mistakes are and how to correct them. We focus on the problem of image captioning in which the quality of the output can easily be judged by non-experts. We propose a phrase-based captioning model trained with policy gradients, and design a critic that provides reward to the learner by conditioning on the human-provided feedback. We show that by exploiting descriptive feedback our model learns to perform better than when given independently written human captions.

Live content is unavailable. Log in and register to view live content