FrontierML: A challenging, end-to-end machine learning benchmark

FrontierML is an end-to-end benchmark that measures how well agentic systems convert monetary capital into improved model performance on realistic, long-horizon machine learning tasks. It gives agents small, fixed wallets ($10, $50, $200), a 12-hour CPU sandbox and minimal validation examples, and lets them control the entire development pipeline: data sourcing, architecture choice, training, compute provisioning and serving. Submissions are required as standalone inference servers that run air-gapped on a single H100 and are evaluated on large out‑of‑distribution held‑out test sets; an automated reviewer agent inspects trajectories for cheating. The suite contains 21 tasks spanning LLM training, efficiency, vision, audio, applied NLP, and games. Compute and synthetic data are available through Spin, a metered CLI service with fixed GPU prices drawn from a price snapshot, and agents can self-check up to four times before final submission.

Experiments show stronger agents reliably improve results when given more budget, but also exhibit inefficient use of time and money: budgets and time were often not fully spent, and agents sometimes innovated new architectures. Leaderboard highlights include Claude Opus (score ~58.8), Claude Sonnet (~50.4), and GPT-6 Astra (~44.5), with human researchers scoring competitively (~56-66 depending on budget). The benchmark emphasizes open‑ended design, robust verification against reward hacking, and realistic constraints to test whether agents can autonomously build production‑quality models as capital scales.