hn.today

Show HN: DeepSeek V3 in ~200 lines of pure Jax

github.com273 points29 comments
Screenshot of Show HN: DeepSeek V3 in ~200 lines of pure Jax

This project implements a compact version of DeepSeek-V3’s architecture in roughly 190 non-comment lines of pure JAX, without relying on higher-level neural-network frameworks. The model combines Multi-head Latent Attention, which compresses what must be cached during generation, with decoupled rotary embeddings and a mixture of experts: one shared expert plus 16 routed experts, of which four are selected per token. It also demonstrates expert balancing through routing biases and an auxiliary loss, as well as an extra prediction head trained to anticipate the following token. The code includes training and cached text generation, with tests for core behaviors such as routing and attention caching.

A default run trained the 36.2-million-parameter model on an Apple Silicon MacBook Air M5 using 207 public-domain detective and crime books. It processed about 98 million tokens over 12,000 steps in roughly 2.6 hours; validation loss fell from 5.02 to 3.51. At step 2,000, balancing biases kept expert usage near 5.7-6.9%, compared with a highly uneven 0.09-18.6% without them, while validation loss was nearly identical. The project presents these results as an architecture demonstration, not a capable language model. Its scale is tiny relative to DeepSeek-V3, and it omits production features including sparse expert execution, FP8 training, long-context support, and post-training. Instructions and released checkpoints enable others to reproduce or inspect the experiment.

Read on github.com29 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in AI

ChatGPT Wallet

ChatGPT Wallet

ChatGPT Wallet lets ChatGPT agents make purchases using tokenized card details, with users reviewing each purchase and approving it through a security check. Launching first for US ChatGPT Pro subscribers, it is built with major payment networks and preserves eligible card rewards and protections. (openai.com)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.