hn.today

Microsoft – more details on the root cause

twitter.com13 points0 comments
Screenshot of Microsoft – more details on the root cause

Microsoft attributes the November 12, 2026, global Microsoft 365 outage to three factors converging on its networking gateway: a latent software defect, increased traffic, and reduced capacity. A gateway-specific code change deployed from July caused lock contention under certain conditions, making servers unresponsive. It escaped testing because test environments used different traffic patterns and generally older, lower-core hardware; the issue was worse on newer, more powerful machines. About 15 hours before the outage, Teams traffic failed over to the primary gateway, increasing its load. Meanwhile, nearly 14% of gateway capacity was unavailable because of an unrelated issue. Neither change caused trouble alone, but together they triggered the defect during peak U.S. traffic. Microsoft says the defect and fix were limited to Microsoft 365 gateway servers, not general Windows deployments.

As servers stopped responding, automatic traffic rerouting overloaded the remaining endpoints, producing a cascade that spread globally after regional capacity partitions failed to contain it. At the worst point, more than 80% of gateway servers were unresponsive. Teams, OneDrive, SharePoint, and Office apps could operate outside the faulty gateway and recovered sooner, while Exchange Online could not. Recovery was prolonged because returning servers were swamped by queued requests, automated recovery was not designed for such widespread failure, and standard rollback mechanisms could not quickly undo a change deployed months earlier. Engineers restored the previous code path by removing groups of servers, updating and restarting them, then gradually returning them to service. Microsoft said a fuller incident review would follow.

Read on twitter.com0 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Other

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.