Processing in Memory: DRAM Is About to Do Math

Samsung has integrated compute units into a commodity LPDDR5X memory package (LPDDR5X-PIM) that exposes 614 GB/s of internal bank-level bandwidth to on-die MAC trees and an ALU, matching high-end unified-memory chips’ bandwidth inside a single 16 GB package. The design places 16 PIM blocks beside banks, streams activations in, reads weights from DRAM into MAC arrays, accumulates partials in a 1-kbit vector file, and returns small activation-sized results to the host. Supported precisions include SINT4/SINT8 and FP8, yielding up to ~2.4 TOPS (SINT4) or ~1.2 TFLOPS (FP8) per package. Samsung demonstrated silicon running Llama 3.1 8B (320-token context) with SINT8 activations/SINT4 weights, showing ~3× throughput (27 → 81.3 tok/s) and ~2.3× runtime speedup versus plain LPDDR5X, and uses an Address Align Mode so conventional DRAM controllers can target PIM without custom hardware.

The core argument is that memory bandwidth, not FLOPs, limits batch-1 autoregressive decoding (GEMV-heavy workloads), and putting math next to DRAM unlocks otherwise wasted internal throughput. Practical obstacles remain: popular k-quant formats (GGUF k-quants) require per-block scaling that minimal DRAM MAC logic cannot perform, physical pages must be bank-aligned (OS and allocator support lacking), and bank-level PIM favors GEMV over narrow GEMM used by grouped-query attention, speculative multi-token verification, and MoE sparsity. Software runtimes, re-quantization tooling, and memory-layout APIs are the main barriers to broad adoption despite compelling silicon and cost incentives versus HBM.