Cerebras Deploys Trillion-Parameter AI Models

Cerebras explains how its wafer-scale systems can run models whose weights far exceed the memory of a single chip, using the open-source Kimi K3, a roughly three-trillion-parameter model, as an example. Under the stated assumptions - 4-bit weights, 8-bit KV cache, and 256,000-token average requests - the weights occupy about 12 terabits, or roughly 16 GB per layer across Kimi K3’s 93 layers. A WSE-3T wafer provides 44 GB of on-chip memory. Because dense-attention layers need extra memory for request-specific key-value caches, the proposed layout puts one dense layer on each wafer; linear-attention layers use a fixed-size state and allow two layers per wafer. The resulting deployment uses 59 wafers for 69 linear and 24 dense layers.

The system runs the model as a pipeline: each wafer holds its assigned layer weights, while only activations travel between wafers over RDMA Ethernet, with about two microseconds per hop. Rather than waiting for one request to traverse the entire pipeline before starting another, different stages process different requests simultaneously; small micro-batches can further balance individual-request speed against overall throughput. The memory left on dense-layer wafers for KV cache sets the limit on concurrent requests, while throughput should be judged by tokens per second per unit of power at comparable generation speeds. Cerebras presents the same memory-mapping and pipelining method as scalable to models larger than three trillion parameters.