GORGO: Online Tuning for Cross-Region Network-Aware LLM Serving

Increasingly, LLM inference services proxy client requests to engine replicas distributed globally. Load-balancing policies must jointly account for factors including KV-cache locality, replica load, and variable network latency when optimizing for metrics like latency and TTFT. However, existing systems only evaluate a subset of these factors in their cost model, leading to uneven concentrations of load and KV-cache across replicas. We present GORGO, a proxy architecture that holistically factors network latency, prefill cost, and queueing delay using tunable parameters. Since open-source chat datasets such as LMSYS-Chat-1M and WildChat-4.8M lack long-context, high prefix-reuse data, we publicly release a synthetic dataset, ART-Chat-2.5M, from long-context production metadata. Intra-user prefix reuse and average token length in ART-Chat-2.5M are 19$\times$ and 6$\times$ higher than in WildChat-4.8M. On a tuning window from ART-Chat-2.5M, evolutionary strategies guide the GORGO policy's parameters to directly optimize p95 TTFT. During a held-out evaluation window, we fix the parameter values learned from tuning and improve p95 TTFT by 15.5\% and p95 end-to-end (E2E) latency by 30.9\% over baseline load-balancing policies such as simple session affinity and prefix-cache.