This project implements a compact version of DeepSeek-V3’s architecture in roughly 190 non-comment lines of pure JAX, without relying on higher-level neural-network frameworks. The model combines Multi-head Latent Attention, which compresses what must be cached during generation, with decoupled rotary embeddings and a mixture of experts: one shared expert plus 16 routed experts, of which four are selected per token. It also demonstrates expert balancing through routing biases and an auxiliary loss, as well as an extra prediction head trained to anticipate the following token. The code includes training and cached text generation, with tests for core behaviors such as routing and attention caching.
A default run trained the 36.2-million-parameter model on an Apple Silicon MacBook Air M5 using 207 public-domain detective and crime books. It processed about 98 million tokens over 12,000 steps in roughly 2.6 hours; validation loss fell from 5.02 to 3.51. At step 2,000, balancing biases kept expert usage near 5.7-6.9%, compared with a highly uneven 0.09-18.6% without them, while validation loss was nearly identical. The project presents these results as an architecture demonstration, not a capable language model. Its scale is tiny relative to DeepSeek-V3, and it omits production features including sparse expert execution, FP8 training, long-context support, and post-training. Instructions and released checkpoints enable others to reproduce or inspect the experiment.
Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.