Hours of YouTube and 85 handtyped pages of notes on LLM Inference – Start here

This is a hands-on teaching project about LLM inference, built around a small C++17/CUDA engine for Llama 3.2 1B Instruct. Its aim is to explain the path from a prompt to generated tokens by implementing the main components from first principles. The learning materials include a video series of more than six hours and over 80 pages of handwritten notes, with topics ranging from loading SafeTensors weights and tokenization to transformer operations, decoding, batching, and paged attention.

The implementation supports Unicode-aware BPE tokenization, GPU weight loading, dense causal prefill across all 16 model layers, and greedy token generation. It also uses a paged BF16 KV cache, with 16-token pages, and implements static and continuous batching so requests can share GPU work while retaining separate cache mappings. A FIFO admission policy reserves enough cache capacity for each request’s stated output budget; requests wait when capacity is insufficient, and oversized requests are rejected. The project remains a fixed-scope learning engine: model dimensions and context length are fixed, prefill handles one prompt at a time, and BF16 rounding means batched and individual runs may produce different tokens. Building it requires an NVIDIA GPU, CUDA, CMake, a C++17 compiler, and the model files. The code is MIT-licensed, but the notes and associated learning materials are reserved.