Bf16 might be freezing your weight updates

The central claim is that fine-tuning model weights directly in BF16 with AdamW can silently discard most optimizer updates. BF16’s spacing between representable values grows with a weight’s magnitude, so a proposed change smaller than about half that spacing rounds away. With Adam steps roughly bounded by the learning rate, this creates a magnitude cutoff: at a learning rate of 1e-6, weights above roughly 4.88e-4 are effectively frozen. The effect acts like a fixed mask that repeatedly trains only a small subset of parameters. Direct BF16 AdamW also makes weight decay a no-op and can prevent its second-moment estimate from shrinking, though the lost updates are the main focus.

A comparison of Mistral-7B checkpoints supports the claim: in Quiet-STaR, trained at a 1e-6 learning rate using BF16 weights and no FP32 master copy, only 5.4% of the 7.24 billion weights changed. Nearly none above the predicted cutoff moved; the smallest weights changed substantially, while most of the model stayed untouched. A rerun likewise found that 97-99.7% of proposed updates vanished at each step, with only 5.6% of weights changing by step 100. The author reports that applying every update with a higher-precision optimizer made the run diverge unless the learning rate was reduced three- to tenfold, and did not produce a better result. This suggests BF16 rounding can both distort what fine-tuning trains and, in some cases, stabilize it by suppressing updates.