A benchmark tests whether frontier language models learned protein sequences themselves or merely absorbed descriptions of famous proteins. It asks models to continue 668 bacterial and archaeal proteins from 167 little-studied families of unknown function, scoring the next amino acid and comparing performance on the original sequence with a shuffled version. The test proteins lack close UniProt relatives, solved structures, and more than three papers per family. Latest Claude models show a pronounced advantage on real sequences: Fable 5.2 is correct about 54% of the time, versus 9% on shuffled sequences; Opus 5.5’s real-versus-shuffled accuracy gap is 27.4 percentage points. Other tested GPT, Gemini, and open-weight language models show little or no gap.
The results suggest that protein sequences formed a substantial part of training data for the newest Claude models. Their accuracy improves with family size, including among families with no papers, and remains high without extended reasoning - patterns consistent with learning from sequence databases rather than scientific text. Fable 5.2 outperforms every tested open protein language model, while Opus 5.5 matches ProtGPT3-10B; both also perform well on famous proteins, where many models can rely on widely available descriptions or copied sequences. This distinction helps explain why models may rank variants of familiar proteins without having learned sequence patterns broadly. The findings concern proteins, not genomes: Opus 5.5 predicts some human and mouse coding DNA but performs poorly on noncoding regions and bacterial genes. The results raise questions about whether this learned sequence knowledge supports structure prediction or variant assessment.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.