Artificial Analysis v5 and the benchmarker's dilemma

Artificial Analysis’s fifth Intelligence Index combines evaluations across four equally weighted areas: knowledge work, coding, science, and general capability. Its subtests include workplace tasks, coding and terminal work, scientific problem-solving, and tests of knowledge and hallucination. The index aims to capture models’ usefulness in current real-world work, but reducing diverse abilities to one score inevitably embeds judgments about which tasks matter and how to weigh them. Because AI capabilities and applications change rapidly, that summary can become misleading or outdated, and rankings may not align neatly with people’s experience of models.

The new index places several closed models ahead of the strongest open-weight model, reflecting a widening gap in frontier flexibility and reliability, especially on science evaluations. The author considers this defensible but warns that a frontier-focused measure could overlook open models’ economic role in cheaper, specialized, privately controlled deployments. Open models may narrow the gap in future cycles, though keeping pace across domains requires strong reinforcement-learning systems and data. The shift toward private test prompts may reduce benchmark-targeted training, but also prevents outside scrutiny and bug-finding. More broadly, benchmark rankings influence labs and markets, yet independent evaluators face conflicts when they sell services to AI companies. The author favors multiple indices with distinct purposes and recognizes a trust trade-off: startups may measure the frontier accurately, while nonprofits, academics, and public institutions may be better positioned to appear impartial.