Evaluating the Early Deployment of a GenAI Co-Pilot for Frontline Maternal and Newborn Health Communication: A Mixed-Methods Case Study from India

Health systems in low- and middle-income countries are beginning to use generative AI (GenAI) to help frontline workers manage high volumes of incoming messages \citep{who2024}. Because these tools often reach patients through human workers, evaluations should examine how GenAI affects workers' judgment, communication practices, and agency, and the services patients receive. A nonprofit health organization and its partners in India operate a two-way WhatsApp service that extends patient and caregiver health education beyond the facility; trial evidence shows such education improves postnatal knowledge and practices \citep{johnston2025}. It receives about 240{,}000 queries per month (18{,}000 medical, 750 emergency) from 80{,}000 users across 15{,}000 facilities. In November 2025, the organization introduced a GenAI co-pilot to a subset of its Medical Support Executives (MSEs), who answer messages in Hindi, Telugu, Punjabi, Kannada, Odia, and Marathi. The co-pilot drafts a language-matched reply to each query from documented patient history and context; the MSE reviews, edits, and sends it. The tool is not patient-facing; its effects run through how MSEs handle its suggestions. The study asks two questions. \textbf{RQ1:} How do MSEs with co-pilot access experience the tool, including changes in attitudes, trust, agency, and practice, and what do MSEs without access expect from it? \textbf{RQ2:} Are conversations handled by co-pilot MSEs measurably different in interactional dynamics, informational content, and rubric-rated quality, and do they match MSEs' reported experiences? The design is concurrent mixed methods \citep{creswell2018}. For RQ1, we conduct 18 semi-structured interviews, nine per group, analyzed with reflexive thematic analysis \citep{braun2022}, paired with interaction logs of each query, AI draft, and sent message; edits to drafts trace where the system meets or misses patients' needs. For RQ2, we build measures that apply to any conversation, with or without co-pilot use. Researchers and clinicians code a stratified sample of about 200 conversations in multiple languages, combining several evaluation rubrics (e.g., clarity, empathy, medical safety, accuracy, next-step guidance) \citep{tam2024} with open coding for patterns the rubrics miss, such as culturally specific markers. We pay special attention to the annotation process because prior efforts report inter-rater agreement below 50\%; documenting it carefully yields methodological insight for AI evaluation in conversational health education, where data are often fragmented and heterogeneous. We then validate an LLM-judge pipeline against human consensus ratings co-developed by clinical experts and medical doctors and apply it to the full corpus \citep{gilardi2023,ziems2024}; clinicians manually review conversations flagged for safety risks to check alignment. We also compute interactional metrics (latency, length, turn count, follow-up) and informational metrics (reading complexity, clinical terminology) from logs. Before launch, 18 India-based MSEs were randomized 1:1 to treatment or control, stratified by language. Once conversations are scored, we estimate whether those handled by co-pilot MSEs differ from control, using multilevel models with conversation-level outcomes, MSE random intercepts, and small-sample corrections for the 18 clusters \citep{kenward1997}. The protocol was approved by the ACE Independent Ethics Committee, Bangalore. For AI evaluation in the Global South, our case study makes two contributions. First, it offers a practical example for evaluating human-in-the-loop GenAI in multilingual settings where ground-truth labels are scarce and outcomes depend on frontline workers' judgments. Second, it also shows how evaluators can triangulate workers' reported experience against behavioral traces in service data, testing whether attitudes toward GenAI match changes in communication.