Apple ML Research: Muse Spark Open with MLX on Mac Studio

This describes how MLX on Mac Studio, leveraging Apple silicon’s unified memory and a new RDMA-over-Thunderbolt driver, enables local and distributed inference of frontier LLMs. Benchmarks compare M5 Ultra and M3 Ultra Mac Studios running three open-weight models: Muse Spark Open (816B total, 42B active in MoE, evaluated in NVFP4), Mistral Medium 3.5 (128B, FP8), and Qwen 3.8 (27B, BF16). Metrics reported include time to first token (TTFT), generation throughput (tokens/sec), and peak power. MLX v0.30.1 adds jaccl collectives for low-latency inter-node communication and provides mlx.launch and mlx_lm.benchmark helpers; an experimental MLX_METAL_FAST_SYNCH=1 flag reduces CPU-GPU sync overhead. Benchmarks use context sizes of 8,192 and 65,536 tokens and show commands and configuration used to reproduce results.

Results show the M5 Ultra’s GPU Neural Accelerator cuts TTFT by up to 4.6× and raises decode throughput by about 1.6× versus M3 Ultra, while peak power remains modest (≈200W M3, ≈300W M5). Qwen 27B and Mistral 128B fit on a single 512GB Mac Studio (≈68GB and 158GB at 65,536 context), but Muse Spark Open requires multi-machine execution (quantized NVFP4 footprint ≈493GB plus intermediates). Pipeline parallelization across two Macs is simple and preserves decode speed while improving prefill throughput for very long contexts (~1.8× at 65,536 tokens); tensor parallelization across multiple Macs yields higher inference throughput (a four‑Mac cluster achieved >36 tokens/sec for Muse Spark Open at 8,192 context) but depends on low-latency RDMA links.