Invited Talk 3 Q/A - Maxine Eskenazi

Most of the work on intelligent agents in the past has centered on the agent itself, ignoring the needs and opinions of the user. We will show that it is essential to include the user in agent development and assessment. There is a significant advantage to relying on real users as opposed to paid users, which are the most prevalent at present. This introduces a study to assess system generation that employed the user’s following utterance for a more realistic picture of the appropriateness of an utterance. This takes us to a discussion of user-centric evaluation where two novel metrics, USR and FED, are introduced. Finally we present an interactive Challenge with real users held as a thread of DSTC9.