Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio

This describes running large open-weight language models locally and across Mac Studio clusters using MLX and RDMA over Thunderbolt. It benchmarks three frontier models - Muse Spark Open (816B total, 42B active, evaluated in 4-bit NVFP4), Mistral Medium 3.5 (128B, FP8), and Qwen 3.8 (27B, BF16) - measuring time to first token (TTFT), generation throughput (tokens/s), and peak power. On a single Mac Studio with M5 Ultra, the GPU Neural Accelerator cuts TTFT up to 4.6× versus M3 Ultra and, thanks to 1.2 TB/s unified memory bandwidth, raises decode throughput about 1.6×. Qwen and Mistral fit on a 512 GB machine (≈68 GB and 158 GB for 65,536-token contexts). Peak power remains modest (~200 W M3, ~300 W M5), enabling desktop deployment for large-context workloads.

Scaling to Muse Spark Open requires multi-node parallelism: MLX supports pipeline parallelization (split by depth) and tensor parallelization (split by width) over jaccl, an RDMA-backed Thunderbolt driver. Pipeline is simple, keeps decode speed similar to one node, and can accelerate large-context prefill by chunking; tensor parallelization can increase throughput but depends on very low-latency RDMA due to many small communications. Using MLX v0.30.1+ and mlx.launch, a four-Mac Studio cluster running Muse Spark Open achieves over 36 tokens/s at an 8,192-token context, demonstrating practical, energy-efficient local and distributed inference on Apple silicon.