AdaSign: Turning 1-Bit Worker Votes into Adaptive Updates

Distributed sign methods reduce communication by sending only the sign of each worker’s local momentum. This removes per-worker magnitude information that could otherwise be used to scale coordinate-wise updates. However, the pattern of signs across workers still carries useful information. For each coordinate, the majority determines the direction, while the vote margin measures how strongly workers agree. AdaSign uses the vote margin to scale each coordinate’s update, assigning larger magnitudes when workers agree more strongly. Under an idealized Gaussian model, we show that the agreement increases monotonically with the momentum signal-to-noise ratio, providing a principled count-to-scale map. In C4 language-model pretraining, AdaSign achieves the lowest validation perplexity among 1-bit worker-sign methods across all evaluated model--budget settings from 60M to 1B. Its practical path uses one model-sized optimizer state, retains $14.1\%$ of FP32 dense-DDP model-sized traffic at 128 workers, and reduces communication-path latency by $49.6$--$55.1\%$ with 64 workers and complete-step latency by $72.1$--$77.2\%$ with 128 workers on P100 GPUs. Together, these results suggest that cross-worker sign agreement can provide useful coordinate-wise adaptivity while preserving the memory and communication benefits of low-bit distributed training.