Microsoft traced a major Microsoft 365 outage to a defect introduced in a July 2026 update for its network gateway servers. The change passed testing and initially caused no problems, but Microsoft’s load tests focused on older, lower-core-count hardware. The defect caused competing processes to contend over connections, eventually exhausting CPU and memory; the effect was worse on newer servers with more cores. On November 12, a Teams failover increased traffic to the gateways while nearly 14 percent of their capacity was unavailable. Though neither condition alone should have exceeded available headroom, together they triggered the defect.
Failures cascaded as traffic moved from unresponsive servers to others, and at the worst point more than 80 percent of gateway servers were unresponsive. Exchange Online could not fail over from the affected infrastructure, leaving many users without Outlook, while Teams, OneDrive, SharePoint, and Office apps could route around it. Restarting servers and adding capacity did not solve the problem, and a conventional rollback was impractical because the change was months old. Engineers ultimately deployed a corrective update that restored the previous code path, gradually bringing services back. Microsoft identified the issue’s cause on November 13 and later confirmed services were healthy. The episode raises questions about production safeguards and whether load testing adequately represented Microsoft’s own hardware and operating conditions.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.