Below One Bit: Training Route and Budget Shape Robustness Under On-Device Storage Constraints

Sub-1-bit weight quantization fits language models into tight on-device storage, yet such models are judged almost entirely on clean text. We ask whether the route to those bits changes robustness to corrupted input. We train byte-level models from scratch (0.86M to 10.03M parameters, six seeds each) and reach the same sub-1-bit format in two ways: either native quantization-aware training (QAT) or post-training quantization of a full-precision model followed by fine-tuning in the format (PTQ+FT). With equal tokens trained in the format, the native model degrades less on Python code in every setting and seed (averaged over nine character corruptions), and again on an independent Python corpus at 4.87M. At this matched budget, PTQ+FT is still the better model on clean and corrupted input. On natural language, the native advantage is far smaller, and no setting in the grid passes a seed-level test. The token budget sets the size and sign of the advantage: quadrupling the native model's tokens reverses it at 0.86M and 2.77M, and in four of six seeds at 4.87M; equal total training tokens leave small residuals of either sign. A PTQ+FT checkpoint nearest the native model in clean quality still degrades more; at the two sizes where the match is close (0.86M and 4.87M), 44 and 30 percent of the advantage remain. On an Apple M5 Max, the packed model is 8.8 to 22.9 times smaller than fp16 but, without a fused sub-1-bit kernel, up to 16.1 times slower at batch 1 than with unpacked weights. Bit width alone does not describe a compressed on-device model; the training route and token budget belong alongside it.