You could have invented transformers from memory

The piece reframes linear layers, MLPs, and self-attention as associative memory stores that hold key-value pairs and are queried by similarity. It centers on a single diagnostic: the interference-to-signal ratio, which measures how much other stored memories corrupt recall of a target value. A simple outer-product (Hebbian) store accumulates memories as W = sum v_i k_i^T and recalls by f(x)=sum v_i (k_i^T x). With random unit keys, interference behaves like a random walk so the ratio scales roughly as sqrt(M/d_k), implying capacity around M ≈ d_k. A gradient descent update on a linear layer is shown to be equivalent to adding such an outer-product memory.

Algorithmic fixes boost capacity. Solving a least-squares problem yields an optimal linear store that whitens keys with (K^T K)^{-1}, canceling d_k directions of interference and raising capacity to about M ≈ 2 d_k; its interference scales like sqrt(M/d_k − 1). Nonlinear recall via MLPs further suppresses weak contributions, and self-attention uses selective (softmax) weighting to act as a nonparametric memory with far higher effective capacity. Numerical examples claim dramatic jumps in practical capacity: with 64-dim keys, from ~64 memories (outer product) to ~2,700 (MLP-like) to over a million (attention-like), illustrating how architecture choices control how much a network can reliably remember.