Long-context and large-batch Mixture-of-Experts training can fail when even one operation briefly exceeds GPU memory, so reducing average memory use is not enough. Four separate peaks can become limiting: expert dispatch, which depends on routing and token loads; vocabulary projection, which scales with tokens and vocabulary size; activations retained at checkpoint boundaries, which grow with model depth and sequence length; and optimizer state, which scales with parameter count. Because the first bottleneck varies with the model, context length, and number of devices, fixing one peak can simply reveal another.
The proposed approach bounds all four peaks by changing when and how computation and data move, while preserving exact losses and gradients. PipelinedLLEP caps tokens per source in expert-dispatch chunks; Ring-DTP streams activation or weight shards through a ring and computes the vocabulary loss with online log-sum-exp; selective checkpoint offload moves a long-lived boundary tensor to CPU memory; and OffloadStreamAdamW pipelines optimizer updates in buckets. Component tests report up to a 59.3% reduction in dispatch memory with no throughput loss, an 86.6% reduction in vocabulary-projection memory, and a 2.05× faster offloaded optimizer step. Combined, the methods enable training 120B-667B-parameter models at one-million-token context, extending context reach up to 32× and achieving up to 10.4× the throughput of a tuned FSDP2 baseline.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.