How I made it possible to pretrain a 1B LLM on 6GB VRAM

A solo experiment demonstrates that a 1.1-billion-parameter language model can be pretrained on a gaming laptop with 6GB of VRAM, challenging the assumption that such work requires a high-end GPU. The training setup runs, saves checkpoints, and resumes correctly, but a complete pretraining run has not been finished; at the measured pace, it would take roughly 132-211 days. Compared with the first working version, optimizations cut each training step from 21.81 to 12.335 seconds and raised throughput from 1,579 to 2,656 tokens per second, while peak GPU use reached 5.24GB. The time estimate varies because the most memory-efficient method reduces model quality.

The central technique trains one model block at a time, keeping only that block’s gradients and optimizer state on the GPU while frozen blocks wait in system RAM. This substantially reduces memory needs but currently produces about 1.6 times worse perplexity than full-model training under the same conditions. Other measures include three-value weights, which shrink saved checkpoints and inference storage but do not reduce training memory; lower-precision matrix operations, which improved speed by about 8% without measurable quality loss; shared input/output vocabulary weights; and recomputing activations instead of storing them. Numerous small implementation improvements account for further speed gains. Mixture-of-experts, automatic ability-change detection, and PyTorch compilation proved unhelpful or counterproductive, underscoring that the results are measured training progress, not evidence of a finished, capable model.