0.5B base model trained on 60B tokens rivals Qwen3.5-2B trained on 30T+ tokens

Nenya is a text-only base model designed to deliver strong pretraining performance with relatively little compute. Its 2.59 billion total parameters include 553 million active per token, and the model card says it was trained on 60 billion tokens. It claims to rival Qwen3.5-2B on pretraining benchmarks while using 500 times less training compute and 3.4 times fewer active FLOPs per token. The architecture combines 120 mixture-of-experts specialists, selecting six per token, with two shared experts and a 528-million-parameter lookup table. Across 32 layers, it mixes Gated DeltaNet-2 linear attention with multi-head latent attention and supports a 16,384-token context.

The evaluation uses likelihood-based scoring through lm-evaluation-harness, with models tested in bf16 using 4,096-token windows; the benchmark sets were excluded from training data. The card also describes custom cloze evaluations and code and maths tests, though the supplied details do not include their numerical results. Nenya is a base model rather than a chat-tuned assistant, so it is intended for text continuation and further adaptation. It can be run through Transformers, packaged kernels, or serving tools such as vLLM and SGLang. Optimized NVIDIA kernels speed up its linear-attention layers; a Mac-specific engine is reported to reach about 125 tokens per second on an M4, while unoptimized execution is slower.