Anthropic warns that it's close to RSI capability, a.k.a. self-building AI

Anthropic’s revised IPO filing signals that it believes AI-driven research may soon reach recursive self-improvement (RSI): a cycle in which AI helps create more capable AI, potentially speeding up progress dramatically. The company says it has already used AI to develop models whose capability gains it did not recognize until after they occurred. Its internal estimates suggest future models may soon meet its policy threshold for “dramatically accelerated progress attributable to AI,” although Anthropic says no current model has crossed that threshold.

Anthropic sees this acceleration as both an opportunity and a safety hazard: AI could boost productivity and scientific discovery, but capabilities might advance faster than the company can test and implement safeguards for safety, alignment, and security. It says it does not expect to let Claude autonomously design and build successive generations of frontier models without meaningful human oversight or understanding, and is considering policy changes to keep people in control. The company’s claims warrant some skepticism because IPO filings can also serve promotional purposes. Still, its reported advances in coding, math, and cybersecurity make its warning significant, and raise questions about whether governance can keep pace with AI development.