The guide explains how to serve GLM-5.3 NVFP4 locally when its 753 billion parameters do not fit on one GPU. Although only 40 billion parameters are active for each token, all the model’s weights must remain available to route tokens to different experts. The roughly 464 GB checkpoint therefore exceeds a GB300 GPU’s 252 GB of memory. The setup connects two GB300-equipped DGX Stations over ConnectX-8 RDMA and uses vLLM tensor parallelism to divide weights and computation between them. Their memory remains separate; both GPUs work on every request, while Station A provides a single OpenAI-compatible API endpoint.
The procedure covers validating the network and storage, downloading the pinned model and copying it to the second Station, checking that both copies match, and launching a distributed vLLM service. Its recommended configuration includes multi-token prediction, FP8 KV-cache storage, and prefix caching to improve throughput and memory use. AIPerf tests on SPEED-Bench Coding prompts reached 607.8 output tokens per second at concurrency 16; at concurrency 32, throughput fell to 385.0 tokens per second and mean request latency was 453.5 seconds. The guide also stresses practical constraints: each Station needs sufficient disk space, access to the gated model and dataset, and a validated high-speed connection. Because the example exposes an unauthenticated API on all Station A interfaces, firewalling and added authentication and TLS are necessary before broader use.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.