SERA: A Multi-Dimensional Evaluation Framework for Retail Agentic Shopping Assistants

SERA: Multi-Dimensional Evaluation for Retail Agentic Shopping Assistants INTRODUCTION Retail shopping assistants are evolving from single-query retrieval interfaces into agentic conversational recommenders that sustain multi-turn dialogue. Users may add dietary or safety constraints, reject recommendations, or revise preferences across turns. Evaluation must therefore extend beyond recommendation accuracy: agents must resist adversarial requests, protect personally identifiable information, comply with responsible-AI and product-safety requirements, and deliver contextually appropriate recommendations. Existing conversational recommender evaluations emphasize preference elicitation and recommendation quality, while many general-purpose LLM-as-judge benchmarks operate on single exchanges. SERA evaluates Safety, Compliance, and Agent Quality jointly at the conversation level. FROM PRACTITIONER REQUIREMENTS TO A METRIC TAXONOMY The SERA taxonomy was derived from requirements gathered across product, legal, responsible AI, trust and safety, brand voice, and recommendation engineering, then consolidated into three evaluation dimensions. Each metric uses an LLM-as-judge prompt with a four-tier rubric, iteratively refined through pilot rounds. The resulting suite contains 21 metrics and supports single-turn and conversation-level multi-turn evaluation where applicable. SAFETY Hard gate / min aggregation • Malicious Attack Refusal • PII Refusal • PII Echo • Sensitive Content Refusal COMPLIANCE Min aggregation • Refusal Appropriateness • Brand Voice & Tone • Conversation Structure • Constraint Adherence • Category Compliance • Uncertainty Disclosure • Disclosure Statement • Reframe Quality AGENT QUALITY Mean by default* • Context Grounding • Answer Relevance • Product Relevancy • Recommendation Decisiveness • Tool Selection* • Tool Arguments Quality • Session Completeness • Correction Recovery* • User Frustration *Tool Selection and Correction Recovery use worst-case (minimum) aggregation. All metrics use a four-tier rubric: 1.0, 0.75, 0.5, 0.0, plus Not Applicable. CONVERSATION-LEVEL EVALUATION For multi-turn conversations, the judge receives the full conversation history and scores each assistant turn in context. Per-turn scores are aggregated using a risk-proportional design: Safety and Compliance use the minimum applicable turn score, so a single non-compliant turn becomes a conversation-level failure; Agent Quality is generally averaged, where occasional imperfection is acceptable. This asymmetry reflects deployment risk: one constraint violation or PII leak can be consequential, while one over-hedged recommendation need not invalidate an otherwise helpful conversation. PERSONA-DRIVEN OFFLINE STRESS TESTING AND PRELIMINARY VALIDATION For offline evaluation during active development or pre-launch, synthetic single-turn seed queries are extended into conversations by pairing the agent with LLM-based synthetic-user personas; only the user side is simulated. Current personas include Shopper, BargainHunter, and ImpatientShopper, with up to five turns. We evaluated approximately 650 synthetic seeds spanning product discovery, compliance, and safety/refusal. Score distributions and qualitative spot checks served as judge-calibration signals: PII Refusal was high and low-variance (mu≈0.97, sigma≈0.07), while Correction Recovery varied more (mu≈0.62, sigma≈0.28), discriminating stronger from weaker agent behaviors. Judge rationales were manually reviewed across score tiers and rubrics were refined iteratively. PRACTICAL VALUE AND NEXT STEPS Used as a continuous evaluation gate during active development, SERA surfaced regressions including relaxed dietary filters across turns, over-refusal of valid product queries, degraded recommendation relevance, and duplicated disclaimers; findings informed system-prompt and guardrail refinement. Validation remains preliminary: persona coverage is narrow, thresholds are not calibrated to downstream business metrics, and formal human-annotation studies, human-LLM agreement analysis, and judge-model ablations remain future work. More broadly, SERA suggests that structured evaluation should guide agentic AI development from the outset rather than serve only as post-hoc validation. Selected references. Zheng et al. (2023), MT-Bench / Chatbot Arena, NeurIPS; Jannach et al. (2021), Conversational Recommender Systems, ACM CSUR; Russinovich et al. (2024), Crescendo, USENIX Security; Li et al. (2024), Multi-Turn Human Jailbreaks.