We need more science in the judgment of AI

The central argument is that judgments about large language models often rely on striking demonstrations rather than scientific evaluation. A result that looks impressive can persuade observers to revise their expectations after the fact, especially when no hypothesis or success criteria were set in advance. The author connects this pattern to practices such as HARKing and p-hacking: exploring results can generate useful ideas, but it does not establish that a capability is reliable or general. Claims about what models can do, or how entire fields will change, need tests designed before the results are known.

Coding demonstrations illustrate the problem. Models may quickly produce familiar games because abundant examples exist in their training data, while performance can falter in less represented areas such as Zig, Nextflow, and biology. The author also questions the idea that LLMs can straightforwardly rewrite bioinformatics software in Rust: in one project, researchers shifted from direct code translation to using models to create a deterministic translation tool, because models missed too many edge cases. Concerns that celebrated scientific results may reflect exposure to researchers’ private chats further complicate claims of discovery. The broader point is that training-data familiarity can masquerade as novel capability, so model evaluations should use rigorous, predefined criteria before supporting sweeping conclusions.