Which Half of Participation Does the Sparse-Averaging Contraction Need?
Abstract
Decentralized training synchronizes model replicas by periodically averaging a random subset of their weights. Existing guarantees assume every replica both contributes to the average and receives the result, although these are separate communication events that fail independently in real systems. We audit this assumption and find an asymmetry: send failures do not affect the leading-order contraction of disagreement, and replicas agree exactly when every replica receives, however few contribute; receive failures reduce the contraction linearly. Send failures instead displace the value replicas agree on, a second channel that doubly stochastic mixing cannot represent because it preserves the mean by construction. The separation gives an effective averaging rate set by the receive fraction, showing the admissible step size is smaller than previously stated, and reveals staleness correction as a finite-time transfer of error between the two channels. All expressions are exact for arbitrary replica weights and validated against a faithful implementation without fitted parameters.