Noise: Speedrun for Pretraining Without Backpropagation

NOISE sets up a constrained competition to explore whether large-scale, non-backprop credit assignment can rival gradient-based pretraining. The rules fix model, data and score: a 37.7M-parameter GPT, a 10M-token FineWeb training set, one epoch limit and final cross-entropy on a validation split, with autograd and any backward passes forbidden. Two tracks measure how low validation loss can go and how fast a target loss (4.2) can be reached. Dust, a method that perturbs activations per token to evaluate many population members in a single forward pass and converts reward-weighted noise into updates for ordinary optimizers, is the baseline. Dust matched backprop with SGD, trails when paired with AdamW, and reported 1,000-10,000x efficiency gains versus EGGROLL-style weight-space evolution strategies.

Community contributions in the first ten days accelerated the speedrun from 1.76 hours to 1.6 seconds (a >4,000x improvement) and drove the best one-epoch validation loss from 4.167 to 3.676, surpassing backprop with Adam in that comparison. Innovations included identity-path credit, orthogonal sampling, Hadamard cycles that probe every neuron, frozen-attention probes, weight-averaging merges, multi-GPU sharding, optimized moment kernels, overlapping communication, and denser update schemes. Organizers plan to raise the challenge (moving from 10M to 30M tokens) and reward optimizer-estimator co-design, with longer-term changes to architectures and hardware expected as work shifts from tuned hacks to scalable algorithms.