A Microsoft software bug that had gone undetected in its data centers since July caused a worldwide outage of Outlook, Teams, Copilot and other services. On Nov. 11, routine traffic rerouting sent Teams users to servers also handling Outlook. Reduced server capacity from an earlier, unrelated problem left the system vulnerable: an unexpected traffic surge exhausted server memory and caused failures. Automated systems then shifted traffic and email to servers they considered healthy, spreading the malfunction. At the outage’s worst point, more than 80% of Outlook’s supporting servers were unresponsive, disrupting work at corporate and government customers and forcing some employees to rely on phone calls and paper.
Microsoft took about 15 hours to recognize the problem, then another 15 hours to identify and begin deploying a fix. Recovery was prolonged because restored servers were overwhelmed by accumulated traffic and failed again; rolling back recent changes did not help because the underlying bug had been present for months. The incident exposed how automated safeguards can amplify a failure and how dependence on one technology provider can disrupt organizations broadly. Microsoft said its testing missed the bug because lab simulations did not reproduce real-world traffic and hardware conditions. In particular, the fault was more damaging on newer, more powerful servers, while tests generally concentrated on older, weaker systems.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.