Beyond Token Counting: Counterfactual Utility Accounting for Agentic AI Workflows

Agentic AI systems consume tokens for planning, retrieval, memory, tool use, delegation, recovery, validation and response generation. Current observability systems measure the token usage, latency, price, and run time of individual tokens [1], and recent research focuses on modelling token consumption patterns [2], costing individual steps [3], and optimizing whole workflows for quality, cost, and latency [4]. These methods, however, do not directly measure the impact of tokens used by each component of the workflow on task success. It is possible that a relatively high-cost span is necessary, but a low-cost span can be of little benefit. We present a framework-neutral approach, counterfactual token-utility accounting, which incorporates semantic span attribution and matched workflow interventions. This method filters, compresses, caches, limits, or replaces that component for a specified range and compares subsequent runs of that workflow to the initial run. It calculates the utility of the token based on the change in quality of the tasks after receiving the tokens, while imposing predefined constraints on the quality and reliability of the tasks. Preliminary validation on a controlled 6 span multi-agent trace results in 9,990 model tokens and 69.1% of the consumption is accounted for by research generation. It also captures up to 840 tokens of uncited retrieval context, retains duplicate calls to a tool for intervention testing, and avoids counting duplicate handoff tokens of 5,200 that would occur. These outcomes illustrate semantic attribution and define potential optimization targets, but are not yet real-world savings. Proposed experiments will involve comparing the proposed method with aggregate token counting and cost-only span ranking, on retrieval, tool-use, and multi-agent tasks. The major contribution is to provide the evidence-based link between observing the consumption of tokens and identifying the reduction in the workflow that is necessary to maintain quality.