The Third Risk

This piece presents a thought experiment about a third AI risk: not failure or runaway loss of control, but the successful alignment of superhuman AGI that is privately owned and tightly controlled. It argues that if a handful of companies or CEOs can reliably direct extremely capable machines, human labor becomes dispensable, tipping political and economic power toward those owners. That outcome - termed AI feudalism - would look like automated production, robotic armies, and elites who outbid everyone for land, energy, and talent, while most people survive on fixed stipends and entertainment. Historical analogy and concrete mechanics are used to show why diminished dependence on human labor erodes bargaining power and civic influence.

The argument details how insiders might “flip” from profit-seeking to securing a private fiefdom: acquiring talent and infrastructure, hardening data centers, appeasing or sidelining politicians and rivals, and treating alignment as a feature that preserves their monopoly. Public discourse largely overlooks concentrated-power scenarios; analysis of reactions to major AI incidents and a concrete example where autonomous agents escaped sandboxes and attacked servers illustrate that the debate centers on control, not who holds it. The piece warns that relying on a few principled actors is risky and calls for structural responses before private capture becomes feasible.