Meta's GB{2,3}00 Cluster

Prometheus is Meta’s gigawatt-scale AI supercluster built at New Albany, Ohio to train next-generation foundational models and ultimately scale to roughly one million GPUs. The first phase is complete and has already trained Muse models, achieving more than 85% goodput on pre-training and over 90% on reinforcement-learning runs, with mean time between interruptions improving from ~47k to over 54k GPU hours after firmware and systems fixes. The build uses closed-loop liquid cooling and mixes traditional and AI-optimized buildings, operating hundreds of thousands of GPUs as a single, tightly synchronized machine with miles of fiber, tens of thousands of switches and millions of optics.

Hardware and network design drive the cluster’s capability and resiliency. Two NVL72 GPU platforms are deployed: Catalina (NVIDIA GB200 NVL72 Ariel) packs 72 Blackwell GPUs across two racks with 192 GB HBM per GPU, ~720 PFLOPS FP8 and 800G scale-out per GPU via paired Grace CPUs and ConnectX-7 NICs; Clemente (GB300 NVL72) raises HBM to 288 GB per GPU, delivers over an exaflop of FP4, native 800G per GPU with ConnectX-8 SuperNICs, and nets a 5-10% performance gain. A dedicated Back-End network and multi-layer building fabrics (initially DSF, now NSF with FBOSS and adaptive routing) plus a multi-Pbps Backend Aggregation layer span up to 100+ km. Reliability is co-engineered with NVIDIA, but the NVL72 72-GPU scale-up domain increases the failure blast radius ninefold, shaping allocation and fault-mitigation strategies.