Quadrillion-parameter transformers and how to train them

The research argues that compute-optimal language models should keep growing when training data is limited, rather than staying at a fixed size as standard Chinchilla prescriptions imply. Its scaling law estimates loss from dataset size, model parameters, training epochs, and ensemble size, accounting for both overfitting and the benefits of ensembling. Extrapolated to 5 trillion tokens and 10²⁹ FLOPs, the prescription calls for more than 300 parameters per token across an ensemble - about 7,000 times Chinchilla’s ratio - with parameter counts continuing to rise as compute grows. The authors present this as a route toward reducing AI’s large sample-efficiency gap with humans.

Experiments show that larger models trained for more epochs need substantially stronger weight decay. The team fits scaling rules for training hyperparameters and reports that these remain near-optimal at 16 times the compute used to fit them, within 0.01 nats of the measured optimum on tested axes. Snapshot ensembling averages predictions from checkpoints in one run, adding no training compute; cyclic learning rates help produce useful, diverse snapshots. Deep ensembles require separate training runs, but become compute-optimal when compute is abundant relative to data, with the best number of members growing roughly in proportion to compute. Combining scaled hyperparameters and ensembling keeps validation loss improving as compute increases, unlike earlier training recipes that eventually worsen.