Evaluating Enterprise RAG Assistants in Production Using Tiered Behavioral Signals

Evaluating enterprise RAG assistants based on static benchmarks of the system’s functionality fails to provide a true measure of the assistant’s functionality within the enterprise’s production environment. To evaluate the enterprise RAG assistant deployed within the company’s internal Slack channels, we designed the Answer Quality Evaluator (AQE). The AQE analyzes interactions between the assistant and users based on a series of behavioral signals in order: user reactions to assistant responses, follow-up messages to the assistant indicating that the user’s issue has been resolved, periods of silence from the users, human intervention within the chat to resolve the users’ issues for them, and finally, the use of LLM-based judgment to resolve the issues in the case that none of the other behaviors lead to the resolution of those users’ issues. By acting in this manner, the LLM is avoided in over 49% of interactions between the assistant and users. The requirements of the system also separate the potential need for human intervention from cases in which the assistant is determined to be unhelpful to the users. These metrics were collected over 5,015 interactions with 25,345 members within 11 different enterprise support channels, with the assistant being helpful in 85.5% of interactions, but with significant variation between channels alone; the lowest being 75.0% helpful in one channel, and the highest being 88.1% in another. Thus, aggregate metrics reveal little of interest regarding the assistant’s functionality overall. Instead, the AQE operates continuously upon the deployed assistant to evaluate its answers live.