Welcome Beam: a 501B open model you can own, customize, and scale

Beam is an open, Apache‑2.0 501B-parameter mixture-of-experts language model designed for coding and agentic workflows that you can run and customize on your own hardware or cloud. It operates sparsely at about 23B active parameters per token, supports text-in/text-out with a 1M‑token context window, and ships with a Rust performance harness plus day‑one support in transformers, vLLM, and SGLang. Checkpoint formats include BF16, FP8, and an NVFP4 format optimized for NVIDIA Blackwell GPUs; the release includes recipes for supervised fine‑tuning and full-parameter RL, a speculative-drafting setup, a response template in the tokenizer, tool‑calling grammar, tunable reasoning modes, and a $10,000 Hub‑credit incentive for community projects.

Architecturally, Beam has 52 decoder layers: three dense SwiGLU layers followed by 49 Mixture‑of‑Experts layers with 256 experts each, routing eight experts per token using a group-limited DeepSeek‑V3 style router; experts hold ~98.4% of parameters so deployment requires full‑model memory while inference cost matches ~23B compute. Attention is hybrid: 43 sliding-window layers (512 tokens) and nine global layers (every sixth and the final) with grouped‑query attention, RoPE in local layers, and a scalable‑softmax query scaler for long contexts. A normalized, depth‑weighted residual stream keeps hidden-state scale constant across layers. Caching and precision choices reduce KV-cache growth and model size (NVFP4 brings BF16 ~1 TB down to ~280 GB), and benchmarks show strong coding, reasoning, and tool‑use performance across precision modes.