hn.today

I had AI read my field

science.caseyhandmer.com14 points6 comments
Screenshot of I had AI read my field

A researcher used AI to audit theoretical astrophysics, beginning with 4,093 papers from Caltech’s TAPIR group and expanding to a citation map of 158,059 papers linked by 554,670 citations. AI screened the full text of 50,269 papers for signs of misconduct and other problems; it found no paper-mill language or hidden chatbot instructions, and only a small number of retractions and unexplained text overlaps. For a deeper check, AI reviewers re-derived equations and recalculated results in 659 influential papers, with a second independent review used to verify flagged errors. The project also maps 32 subfields and assesses 155 open questions for whether AI could help answer them.

The audit verified 872 errors in 433 papers; 159 errors were copied into later work. Of the 872, only 47 had been criticized in print, while 367 papers copied an error and just 68 criticized one. In a close review of 200 papers, 13 errors had been removed between preprint and publication, but 56 remained in journal versions, suggesting peer review caught only a minority. Errors also persisted: half were still being copied seven to nine years later, and most verified errors in the 200-paper sample remained in current preprints. The author estimates roughly 34,000 errors may exist in highly cited papers, but the intensive audit covers only 4% of those papers. The project cost about $2,720 in model usage, and its findings remain limited by incomplete coverage and the need for human scrutiny.

Read on science.caseyhandmer.com6 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Science

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.