A researcher used AI to audit theoretical astrophysics, beginning with 4,093 papers from Caltech’s TAPIR group and expanding to a citation map of 158,059 papers linked by 554,670 citations. AI screened the full text of 50,269 papers for signs of misconduct and other problems; it found no paper-mill language or hidden chatbot instructions, and only a small number of retractions and unexplained text overlaps. For a deeper check, AI reviewers re-derived equations and recalculated results in 659 influential papers, with a second independent review used to verify flagged errors. The project also maps 32 subfields and assesses 155 open questions for whether AI could help answer them.
The audit verified 872 errors in 433 papers; 159 errors were copied into later work. Of the 872, only 47 had been criticized in print, while 367 papers copied an error and just 68 criticized one. In a close review of 200 papers, 13 errors had been removed between preprint and publication, but 56 remained in journal versions, suggesting peer review caught only a minority. Errors also persisted: half were still being copied seven to nine years later, and most verified errors in the 200-paper sample remained in current preprints. The author estimates roughly 34,000 errors may exist in highly cited papers, but the intensive audit covers only 4% of those papers. The project cost about $2,720 in model usage, and its findings remain limited by incomplete coverage and the need for human scrutiny.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.