Unified GPU Memory for MoE Inference: Online Reallocation Between KV Cache and Expert Weights

Mixture-of-Experts (MoE) models reduce inference compute by activating only a subset of their experts, but serving them requires GPU memory for both key-value (KV) cache state and expert weights. Existing systems reserve fixed capacity for each resource, even though the right balance changes with the workload. We present a unified GPU memory pool that makes KV pages and expert weights interchangeable occupants of one paged buffer: either resource can reclaim capacity from the other at runtime, with the pool reorganizing resident pages as needed to satisfy each type's layout requirements. The allocation of capacity between KV cache and expert weights becomes a runtime variable rather than a startup constant. We drive the pool with a deliberately simple mixed-LRU policy, isolating the benefit of the shared address space from that of the policy. On OLMoE-1B-7B-Instruct, the pool matches the best static configuration on stationary workloads without manual tuning, and achieves a $1.48\times$ speedup in total execution time when the workload shifts from KV-heavy to expert-heavy.