Fast Forward: Accelerating On-Device LLM Prefill With Predictive FFN Sparsity

The prefill stage of large language model (LLM) inference is a major computational bottleneck on resource-constrained devices, where feed-forward networks (FFNs) dominate compute at short-to-moderate context lengths. We present \textbf{FastForward}, a predictive FFN sparsification framework for efficient prefill: a lightweight context-aware expert predictor selects important neurons per block, an error compensation network preserves accuracy under sparsification, and a layer-wise sparsity scheduler allocates budget by layer importance. Across LLaMA and Qwen models up to 8B parameters, FastForward achieves up to 1.82$\times$ measured FFN-module speedup and up to 1.45$\times$ projected compute-bound end-to-end prefill speedup at 50\% FFN sparsity, limiting LongBench degradation to under 6\%. At matched 50\% sparsity, FastForward outperforms CATS by 9 - 12 points and a GRIFFIN-style first-block-static baseline by 6 - 13 points in LongBench average across model sizes.