The Emerging Science of Machine Learning Benchmarks

This work examines the role of benchmarks as the engine of progress in machine learning and pinpoints why they both succeed and now face a crisis. It lays out the mathematical foundations - treating populations as probability distributions and showing that sample requirements grow quadratically with the inverse of the effect size - and explains the holdout method as the discipline that separates unconstrained training from strict testing. It traces how repeated adaptive use of test sets undermines statistical prescriptions, why a broader scientific replication crisis applies to ML, and why model rankings nonetheless reproduce more reliably than raw accuracy numbers. Social norms and community practices, not just formal statistics, are credited with staving off collapse. The shift to internet-scale generative models and post-training alignment creates new evaluation challenges that traditional benchmarks were not built to handle.

The work catalogs concrete problems and remedies: the dangers of test-set reuse, the subtle importance of labeling and annotation, the limits of aggregating tasks (analogous to voting systems), and performativity - where deployed models change future data and break evaluation. It proposes dynamic benchmarks and discusses frontier issues as human supervision becomes the bottleneck and automated mutual evaluation has inherent limits. Overall it offers diagnostic findings, precise technical points, and prescriptive directions for adapting benchmarking practice to modern, large-scale AI.