Investigating the effect of advertisement in AI chatbots on users

This study examines how advertising in large language model (LLM) chatbot interfaces changes user perceptions of both the chatbot and the advertised product. In a single-session experiment, U.S.-based participants are randomly assigned to one of four conditions: (1) a short conversation with an AI chatbot containing no advertisement, (2) a short conversation with an AI chatbot containing a generic advertisement, (3) a short conversation with an AI chatbot containing an advertisement coupled with a personalised message from the chatbot about it, or (4) reading an article containing a generic advertisement. In a pre-survey, participants answer questions about how often they use AI chatbots, their trust in AI chatbots in general, their attitudes towards online advertising, and their privacy concerns; we use these as moderators. In a post-survey, participants answer questions about the fit of the advertised products, their perception of personalisation, their ability to recall the products, and their trust in the chatbot; with each measure administered in the conditions where it applies. In three pre-registered hypotheses, we predict that both the medium (chatbot versus article) and the personalised message from the chatbot (chatbot with message about ad versus generic ad) will impact participants' perception of the advertisement as a good fit for them and as more personalised to them; we additionally explore effects on ad recall. We also predict that the presence of ads in chatbots (no ad versus generic ad) and the presence of a personalised text about this ad (generic ad versus ad with message from the chatbot) both lead to a decrease in trust in the chatbot. We test each prediction with a pre-registered linear regression on the two conditions it compares, correcting the p-values within each hypothesis for the two comparisons it makes. For that, we recruit 2,000 participants, 500 per condition, a sample size set by a power analysis for 82% power to detect an effect at Cohen's d = 0.20. The experiment is pre-registered and has been approved by our institution's research ethics review. At the NeurIPS workshop, we would like to present the results of this experiment and discuss policy implications and technical follow-up work.