A study of sequence weighting at scale

The study examines how varying the loss weight of individual training sequences changes what language models learn as they scale. Each sequence is assigned a random weight drawn log-uniform between 0.01 and 10, models are trained (JS-dense, JS-sparse, and Qwen 2.5 spanning ~500M to hundreds of billions of parameters) for three epochs, then re-evaluated on the training set. An effective sequence weight exponent p is fit to describe how loss reduction on a sequence scales with its weight (p=0 means weight has no effect, p=1 means loss reduction is proportional to weight, p>1 means high-weight sequences dominate). Hyperparameters are tuned on validation data or extrapolated with Chinchilla-style scaling for the largest models.

Results show a consistent non-monotonic pattern: p is low for small models (they learn general patterns irrespective of weight), rises for medium-sized models (they concentrate capacity on idiosyncratic, higher-weight sequences), then falls again for very large models (they can learn nearly all patterns, again diminishing weight-dependence). Training for more epochs shifts the p peak toward smaller models, and absolute p values vary across families. These "aberrant" scaling behaviors complicate naive hyperparameter extrapolation; suggested responses include evaluating mixes only at sufficiently large scales, adjusting sequence weights to compensate, and continuous measurement to detect breakdowns in scaling assumptions.