From Conversational Chats to Agents: Workload-Dependent Bottleneck Shifts in LLM Serving

LLM serving systems are commonly evaluated using independent conversational requests, yet emerging agents generate long-lived workflows with dependent model calls, tool-induced pauses, and repeatedly growing context. We ask how these workload changes shift the dominant serving bottleneck and, consequently, which serving optimization remains useful. We study two complementary mechanisms against a common vLLM baseline: QLM's queue-aware scheduling under ShareGPT workloads, and Continuum's cross-turn KV-state reuse under multi-turn HyperAgent traces. Across conversational workloads varying arrival rate, burstiness, concurrency, and prompt length, QLM reduces p99 scheduling delay from 151.2 to 47.4 ms at moderate load and from 110.4 to 50.2 ms for long prompts, while throughput improvements remain modest. Under severe overload, both QLM and vLLM saturate near 4.8 requests/s and their queues grow almost identically, showing that scheduling cannot compensate for exhausted GPU capacity. Agentic traces expose a different bottleneck: repeated prefill of conversational history. Continuum eliminates up to 2.7M redundant prefill tokens on a single trace and reduces cumulative completion time from 162 to 105 minutes, a 35% reduction. Together, these results reveal a workload-dependent bottleneck shift from queue ordering, to raw serving capacity, to repeated state reconstruction, motivating adaptive ML-serving policies that select scheduling, capacity, and state-retention actions based on observed workload conditions.