Implementing GPT-2 in CSS

Shubham Patil describes building a webpage that generates text with GPT-2 Small using CSS alone: no JavaScript, WebAssembly, or runtime code in another language. The roughly 124-million-parameter model produces a token in about 15 seconds. Early experiments used CSS calculations to reproduce neural-network operations, including normalization and attention softmax, but failed when the browser expanded nested custom properties into enormous expressions. Registering properties with CSS’s @property rule made the browser calculate and retain numeric values, allowing a tiny test transformer to match Python’s predictions. Scaling that approach directly to GPT-2, however, would have required an estimated 3-6 GB of stylesheet, so Patil looked for a more compact way to do the matrix arithmetic.

The key insight was that variable fonts already compute dot products. A font’s glyph widths can vary with multiple axes, and the browser combines each axis value with stored width-change deltas - effectively multiplying a matrix by a vector. A test font represented a 64-by-64 matrix in about 8.9 KB, with errors under a quarter of a font unit. For GPT-2’s 768-dimensional inputs, Chromium’s limit of 256 axes per font required splitting each matrix across three fonts; placing their glyphs together lets the browser add the partial results. CSS supplies the model’s values as axis settings, while scaling factors compensate for font-format limits and integer-only deltas. Patil also describes balancing weight precision against glyph-width limits, noting that 8-bit weights introduced too much error and 10-bit weights could exceed those limits.