How the bad science of AI doomerism is good for big business

The piece argues that alarmist claims that current AI systems will inevitably reach superintelligence and threaten human extinction lack scientific grounding and transparent justification. High-profile resignations from Anthropic and OpenAI, and the prominence of effective altruism and longtermism in shaping the doomsday discourse, have amplified vague probability estimates of “p(doom)” - figures like Dario Amodei’s 25 percent or Evan Hubinger’s >10 percent - without published data, methods, or reproducible calculations. Critics including Emily Bender and Boyan Milanov call these numbers a parlor game rather than testable science, and trace the rhetoric back to cultural movements that cast advanced AI in transcendent, god-like terms.

Those narratives serve powerful commercial interests by generating fear, attracting investment, and giving leading firms leverage to dictate safety standards and regulatory pacing. Company leaders press for industry-led slowdowns and embedded evaluators - Dario Amodei’s proposed plan - positions framed as prudent but that critics like Timnit Gebru say tailor regulation to incumbents’ advantage. Questions also surround industry-linked evaluators such as METR. Meanwhile concrete harms - environmental costs, labor displacement, privacy degradation, surveillance, creative theft, and documented near-misses like an AI-influenced military misstep - are already material and require independent oversight, yet are being overshadowed by speculative extinction narratives that consolidate corporate power.