Skip to yearly menu bar Skip to main content


Modal

Expo Talk Panel

Any GPU, Any Cloud: Training on a Planetary Fleet

Pyrmont Theatre
[ ]
Sun 6 Dec 11 a.m. AEDT — noon AEDT

Abstract:

Modal operates a large GPU fleet spanning 20+ clouds around the world: the antithesis of your traditional homogenous training cluster. This talk covers two directions of research along these lines. For RL, we distribute rollouts across regions and GPU types, using delta compression to keep weight sync cheap over slow links. For pre-training, we introduce a streaming DiLoCo algorithm with cloud-topology-aware hierarchical updates across heterogenous compute.

Chat is not available.