Model selection for Hindi form-filling voice agents under noisy speech, cost and latency constraints

We describe how we evaluated a voice agent, now being piloted in India with lowincome, Hindi-speaking mothers, that fills a structured maternal-care enrollment form over a phone call. A wrong value captured or a call ended early means a mother is not enrolled. Those stakes necessitate pre-deployment testing and rule out building an evaluation dataset from real callers’ data, which is personally identifiable. We therefore built our own: native Hindi-speaking women re-recorded failure modes observed across dozens of pilot calls under the noisy conditions callers speak from, giving 380 recordings assembled into 960 simulated calls and 3,760 turn-level tests drawn from those multi-turn calls. We score each component in isolation and in an end-to-end pipeline to separate the errors a component makes from those it inherits. The best model extracts almost perfectly from reference transcripts (99.79%), but median form completion falls to 85–92% on real transcripts. Per-turn evaluation also ranks models incorrectly: under the deployed transcription model, the model ranked 10th per turn completes more form fields than the model ranked 3rd because some of its errors are recoverable by rules. This lets smaller, cheaper models match larger ones on form completion. Finally, no model wins on accuracy, cost and latency at once. We show how we make that trade-off when choosing the models we deploy.