Stable Counts, Shifting Signals: A Longitudinal Stress Test of Enterprise GenAI Telemetry
Perla Molina ⋅ Isabel Metzger ⋅ Zach Whitman ⋅ Richard Speidel
Abstract
Enterprise generative-AI (GenAI) deployments are often monitored through snapshot counts, yet longitudinal comparisons can conflate changes in use with model migration, denominator choice, annotation drift, and multi-label network expansion. We present a privacy- and provenance-aware audit for testing whether enterprise GenAI telemetry remains comparable across time. The method compares adjacent periods at multiple levels of granularity, separates scale from per-conversation resource measures, quantifies distribution shift, stratifies label and edge stability by support, and diagnoses combinatorial inflation in transition networks. We evaluate it on 65,499 de-identified conversations from 15 anonymized organizational units in a public-sector deployment. The two quarters contained similar numbers of conversations: raw volume increased 0.8\%, while conversations per calendar day decreased 0.3\%. Meanwhile, model responses per conversation fell 5.2\%, estimated tokens per conversation rose 2.6\%, and estimated cost per conversation fell 15.4\%. Organizational composition was stable (rank $\rho=.993$; total variation $=.023$), as were provider shares (total variation $=.013$), while model-version composition shifted sharply ($.550$). Overall overlap was low for open-vocabulary labels (Jaccard $=.273$) and directed transition edges ($.121$), but substantially higher for their top-20 sets ($.739$ and $.818$). Mean labels per conversation increased 17.5\%; transition pairs per conversation increased 36.8\%, close to the 38.2\% first-order increase implied by quadratic multi-label expansion. These results show that stable adoption counts can hide both system change and representation-induced artifacts. We recommend provenance-aware, support-aware, and multiplicity-aware reporting before interpreting telemetry as workflow change.
Chat is not available.
Successful Page Load