hn.today

Astra 6.2 and Sol 6.2 flop at Simple Bench

simple-bench.com159 points155 comments
Screenshot of Astra 6.2 and Sol 6.2 flop at Simple Bench

SimpleBench is a multiple-choice benchmark designed to test abilities that its creators say remain difficult for frontier language models: spatio-temporal reasoning, social intelligence, and resistance to linguistic tricks. It contains more than 200 questions and focuses on familiar, everyday reasoning rather than specialized knowledge. The benchmark’s central argument is that strong performance on conventional tests does not guarantee reliable answers to basic questions, because models can depend on memorized knowledge and approximate reasoning that fail when wording or context is challenging.

The reported results put a small human sample ahead of every tested model: nine participants averaged 83.7%, while the top-scoring model listed in the introduction, Claude Fable, reached 81.9%. Model scores are averaged across five runs, with temperature set to 0.7 and top-p to 0.95 except for the o1 series; the team also uses a prompt engineered to improve benchmark-specific performance. The page directs readers to a technical report for details on methods and human evaluation. Although the headline says Astra 6.2 and Sol 6.2 flop, the supplied content gives no individual scores or rankings for those models, so it does not show how they performed.

Read on simple-bench.com155 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.