Lithos-metal: Qwen3.8-27B on 40-core M5 Max at 168 tokens/s

Lithos-metal is a fully open-source inference engine that generates Metal megakernels to run low-latency large-model inference on Apple silicon. It provides a local serving CLI (installable via a Homebrew tap) with a default HTTP API, OpenAI/Ollama-compatible endpoints, and client launchers for coding agents (opencode, claude, codex, hermes). Targeted hardware requires macOS 26+ and enough unified memory for target, draft, and context; the Quick Start shows launching a Qwen3.8-27B NVFP4 server and connecting clients without changing global configuration. The server downloads model/draft pairs, prepares cached weight packs, compiles kernels, and exposes streaming chat completions, responses, and tool-call adapters.

The engine compiles operators into GPU tasks and uses layer-wise fusion to create mixer megakernels (Gated DeltaNet, full-attention, or DSpark mixer) while keeping MLPs separate, executing persistent Metal threadgroups that retain local intermediates and GPU-resident state via an indirect command buffer. DSpark is a speculative block drafter plus a sequential Markov head that lets one target pass commit multiple accepted tokens, amortizing weight reads. Fusion and kernel tuning exploit Apple matrix accelerators, dynamic on-chip memory, and local reuse. Measured on a 40-core M5 Max with 48 GB, a Qwen3.8-27B NVFP4 target compute projection reaches ~168 tokens/s at 128-token context and ~127 tokens/s at 32K context (assuming six accepted tokens per eight-row target pass); timings exclude drafting and acceptance overhead.