A false smoke alarm almost drowned Jane Street's only NYC datacenter

Jane Street kept its New York datacenter inside its office building, close to the trading floor, partly so staff could physically cut power to runaway trading systems. The room eventually held roughly ten rows of server cabinets, a UPS, cooling units, and backup power. A contractor’s work nearby stirred up dust and triggered the smoke alarm. The alarm shut down the computer-room air conditioners and pressurized the sprinkler pipes; although sprinkler heads would release water only when heated, the servers’ heat rapidly raised the room’s temperature. With the datacenter at risk of overheating until the sprinklers activated, a nearby systems engineer raced in and unplugged the equipment cabinet by cabinet. The temperature peaked around 143°F, then stabilized, and the firm restored service the next morning, with little disruption because markets were closed.

The incident exposed how protective systems can interact in dangerous ways: a false alarm meant to prevent a fire also removed cooling, while the sprinkler system turned the resulting heat into a threat to the entire datacenter. In follow-up discussions, the team kept the sprinklers and the office-based datacenter, opting instead to improve procedures and monitoring. The engineer also wrote an OCaml monitor that queried Linux servers’ onboard temperature sensors every minute, averaged the readings, and alerted the systems team to a broad rise in heat. Fleet-wide server data was considered more dependable than existing cooling telemetry or room sensors, and could reveal a general cooling failure before individual machines produced useful warnings.