Deep dive into Azure Maia 300 architecture

This explains the architecture and engineering trade-offs behind Azure Maia 300, a system-level platform engineered to maximize inference throughput and efficiency for long-context, decode-heavy, agentic LLM workloads. Maia 300 uses a software-defined locally accessed dataflow (SDLA) that separates control from data execution so software explicitly manages placement, movement, conversion, and synchronization across local SRAM, DMA engines, HBM, compute, and the network. The chiplet-based SoC places compute on TSMC N2P while moving memory-controller and I/O to dies optimized for bandwidth and cost, boosting compute FLOPs by 36%, increasing scale-up bandwidth ~14%, and delivering 2.5× the memory bandwidth versus the prior generation. Each accelerator includes eight HBM4 stacks (288 GB, 18 TB/s per accelerator), software-managed on-die SRAM, ISA and vector-engine enhancements for key inference kernels, and new narrow formats (MXFP4 and sub-3-bit) plus hardware-accelerated type conversion to shrink memory traffic and support larger models and longer contexts within the same power envelope.

Networking and deployment are treated as part of the same fabric: racks of 96 accelerators form pods with a half-switched in-rack topology and optical second-tier switches that scale to superpods of up to 12,288 accelerators. Integrated AI transport gives a single communication model across the superpod, with three latency domains (direct pair, one-hop pod, three-hop cross-pod) and per-job bandwidth allocation (an accelerator can drive up to 750 GB/s to a peer under favorable packing). That design enables flexible job placement, independent prefill/decode workers with direct KV transfers, hot-spare resilience across racks, and a focus on maximizing useful tokens per watt and per dollar at datacenter scale.