Mass attempted replication of old findings of LLMs with new models

A large-scale replication study tests whether published findings about LLM behavior still hold as models improve. Researchers revisited 50 widely cited claims across 20 models released between July 2024 and September 2026, spanning eight developers and models of different sizes and openness. They repeated each claim’s original test where possible, tracked how effects changed with model release date, and classified findings as expired, weakened, persistent, or strengthened. An effect counted as expired when no more than one-fifth of its original size remained. The study finds that 44% of tested claims had expired by September 2026, 34% had weakened, 16% persisted, and 6% strengthened. Examples of faded capability failures include fabricated citations, basic counting and arithmetic errors, difficulty retrieving information from long contexts, and abandoning correct answers when challenged. In contrast, some social and judgment behaviors - such as flattering users and judges favoring their own answers - persisted or grew stronger.

The estimated average lifetime among the 22 claims with expiration dates was 15.7 months, with a median of 14.7 months. Across all claims, an estimated 16% had expired within a year of publication and 39% within two years; the share still near its original strength also fell over time. Results varied by topic: reasoning and sycophancy claims often expired, while some biases and judgment effects lasted longer. These are fitted trends across a model panel, not direct measurements of one newest model, and the study relies on minimal test designs, limited early model cohorts, and substitutes where original models were unavailable. The findings suggest that claims about LLM behavior need regular retesting rather than being treated as lasting facts.