Hit Rate Is Not Value: When Cache Metrics Mislead in Tiered LLM KV-Cache Persistence

We set out to beat LRU at evicting persisted LLM KV caches at different tiers (GPU VRAM, NVMe, Object storage) with a learned policy and failed at this. Across three synthetic workloads, two served architectures and a replay of the Azure LLM inference trace, LRU wins at cache hit rate every single time. However, if we measure in terms of GPU time actually saved, LRU delivers the least value of any policy we test on enterprise level of traffic. Here, a value density policy concedes 21 hit rate points to LRU by saving 1.99 times as much compute and still 1.15 times once we correct the architecture the same analysis exposes. Hit rate assumes every hit is worth the same, but KV caches vary in size and compute time. A hit below the context length N* is worth nothing, and above it the savings grow superlinearly. Both effects depend on N*, and the largest factor they expose is provisioning, not policy: deleting a tier that cannot justify the restore time is worth 3.6 times at identical capacity. A ground truth ablation study isolates the failure where two clairvoyant policies that both cache perfectly with 100% hit rate still deliver significantly different value in terms of GPU compute time.