From Calibration to Income: A Four-Level Framework for Evaluating AI Systems in Global South Deployments

AI-for-development teams routinely report that a model "works," but that claim usually rests on one number, such as a benchmark accuracy score or a satisfaction rating, that says nothing about whether anyone's circumstances changed. We report field experience applying a four-level evaluation stack consisting of model performance, product engagement, behavior change, and development-outcome impact to an AI-enhanced WhatsApp Upskilling System (WUS) for youth livelihoods deployed with young adults in West Africa. Applying a shared framework proposed for AI-for-development practitioners, we instrument each level with one interpretable metric: an LLM-judged scripted-scenario pass rate (0.52), a weighted engagement index (0.45), a six-behavior change index (+0.33 behaviors against a screened control), and a baseline-adjusted employee-earnings gain (0.25 SD, 95\% CI [0.03, 0.47]); the last two come from a program-level panel and are not causal estimates of the AI component. The levels differ in sample, design, and horizon, and reveal different problems: adequate technical performance coexists with a rapid collapse in engagement, and a single engagement average hides a concentration (Gini~=~0.73). We argue the stack's value is diagnostic rather than evaluative: a weak result at one level routes attention to a specific, fixable cause rather than a verdict that "the AI failed," and we report where that diagnosis already changed what we are building next.