Workshop on Evaluation of Interactive Agents

The rapid transition from large language models (LLMs) as single-turn assistants to interactive agents has created an urgent need for new evaluation methodologies. LLMs are increasingly deployed in high-impact settings such as education, counseling, negotiation, research assistance, and software development, where success depends not only on generating a correct response, but on sustaining effective interactions over extended trajectories. Modern agents must infer user goals, ask clarifying questions, maintain context over many turns, use tools, recover from mistakes, and collaborate with users and other agents. The rise of agentic AI systems and multi-agent workflows has made these evaluation challenges particularly timely. Traditional static benchmarks can miss whether an agent adapts over time, asks useful clarifying questions, recovers from earlier errors, or remains reliable across repeated trials. As conversations extend, model performance can degrade, disparities across user groups can become more pronounced, and stochastic agent behavior can make a single run misleading. At the same time, evaluating interactive agents directly with real users is slow, expensive, difficult to reproduce, and hard to scale in expert domains. This has led to the growing use of LLM-based user simulators, where another model emulates user behavior or task dynamics to support evaluation, training, and stress testing. However, existing simulators may fail to preserve latent user states, reflect diverse human attributes, represent realistic goals, or match the interaction style of real users. These challenges call for a dedicated venue on evaluation methodology for interactive agents.