Grace Blackwell needs a new scheduler: 27% more usable GPUs from the same racks

Large AI training and inference are constrained not only by GPU count but by communication speed. Tensor parallelism requires repeated all-reduces, while mixture-of-experts models use all-to-all exchanges; moving these operations from NVLink to InfiniBand can sharply reduce throughput. NVIDIA’s NVL72 rack connects 72 GPUs through a non-blocking NVLink fabric, making multi-node communication within a rack much faster. That helps training jobs keep large expert-parallel groups on NVLink, and lets inference systems spread model weights across more GPUs to free memory for larger KV caches and higher request throughput. But jobs that cross rack boundaries still encounter slower networking, so realizing the hardware’s gains depends on where workloads are placed.

Rack-scale scheduling also has to account for failures and unevenly freed capacity. Treating a full 18-node rack as one allocation unit makes any node failure disrupt the entire rack; the cited reliability model estimates a rack interruption about every 27 days. Leaving two nodes per rack available as spares can absorb early failures. For a 128-rack cluster with roughly 2% node unavailability, 16-node slices yield 2,037 usable nodes, 27% more than full-rack 18-node slices, and 4% more than 9-node slices. The proposed scheduler approach combines topology-aware placement, replacing failed GPUs in place, and automatic cluster defragmentation so multi-node jobs stay rack-local even as failures and job completions scatter available capacity.