Microsoft attributed the November 12, 2026, global outage affecting the networking gateway behind many Microsoft 365 services to three factors acting together. A July software change had introduced a latent lock-contention bug that could make gateway servers unresponsive under particular traffic conditions. Testing missed it because test environments used different traffic patterns and generally less powerful hardware. About 15 hours before the outage, Teams traffic shifted from an alternative gateway to the primary gateway, raising load and changing traffic patterns. At the same time, an unrelated issue had left nearly 14% of gateway capacity offline. Neither change alone was considered enough to cause an outage, but together they triggered the bug during peak U.S. traffic.
As servers failed, automatic traffic rerouting concentrated requests on the remaining servers, spreading the failure across the fleet; more than 80% became unresponsive within two hours. A separate problem prevented the usual isolation boundaries between capacity partitions from containing the cascade, allowing it to spread globally. Teams, OneDrive, SharePoint, and Office apps recovered sooner because they could use other gateway infrastructure, while Exchange Online could not. Recovery was prolonged as returning servers were overwhelmed by queued traffic, and standard rollback tools could not readily reverse a months-old change. Engineers restored the previous code path by updating and restarting groups of servers in stages. Microsoft said a fuller incident review would provide further details.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.