LatentRoof: Transferable Latency Prediction Across AI Accelerators

Modern ML workloads run across heterogeneous accelerator landscapes, including GPUs, TPUs, and emerging AI chips. Differences in their architecture make the same workload perform differently across devices, making accelerator selection a challenging problem. Manual kernel engineering requires per-accelerator expertise; LLM-based kernel generation requires compiling and profiling the workload on each target device. As accelerator diversity increases, repeatedly profiling workloads on every device becomes increasingly expensive and difficult to scale. We propose a hardware-aware hybrid performance model that predicts end-to-end workload latency on unseen accelerators without execution. An analytical model estimates ideal performance from workload compute and memory demands together with hardware specifications, while a learned model captures workload-specific behavior and residual overhead. Since, hardware enters through analytical model, prediction transfers across unseen accelerators. Preliminary results on three GPUs yields 12.2–17.6% MAPE for operator-level prediction on unseen GPUs across 11 workload families, and 9.5–17.1% MAPE for end-to-end GPT-2/BERT inference on seen GPUs.