An Open Field Guide to Inference Engineering

This field guide explains why platform engineers are increasingly responsible for running AI models themselves and how to approach the work. Open models can now compete with closed services, while self-hosting offers control over latency, reliability, and data location, and may lower costs substantially at high usage. It is most compelling when hosted-model bills become significant, vendor performance no longer meets requirements, or sensitive data must stay inside an organization. The guide frames inference engineering as a form of SRE: familiar skills in scheduling, capacity planning, load balancing, autoscaling, rollouts, observability, and FinOps still apply, but must account for GPUs, model behavior, and long-lived requests.

The technical core is sizing and serving models efficiently. An inference engine loads fixed model weights into GPU memory and maintains a growing KV cache for active conversations; cache use rises with context length and concurrency and can become the main limit on users per GPU. For example, a 70B model can require roughly 40 GB of cache for one 128K-token conversation, while runtime overhead means a model that loads successfully can still run out of memory later. Requests have two distinct phases: prefill is compute-bound and shapes time to first token, while decode is generally memory-bandwidth-bound and shapes output speed. The guide also introduces a four-layer stack - kernels, engine, control plane, and gateway - and stresses tracking user-visible latency and throughput separately.