Architecture Decisions for Marin 535B-A23B

Marin 535B-A23B completed pretraining on 18 trillion tokens on November 5, 2026. Its design pairs 535 billion total parameters with 23 billion active per token, using 48 layers, a 6,144-wide hidden dimension, and eight latent mixture-of-experts layers with shared experts. Attention alternates three local sliding-window layers for each global layer; other notable choices include gated normalization and attention, short convolutions, quantile-based expert balancing, and a MuonH-centered optimizer. The team prioritized realized hardware utilization and predictable training over serving efficiency, while aiming to finish by December and leave time for initial reinforcement-learning work. Earlier Marin MoE runs had met their loss forecasts without spikes, so the team favored designs that preserved that stability evidence.

Pre-launch testing surfaced failures that standard checks missed. A sharded GPU norm calculation could abruptly shrink the language-model output weights and spike loss; applying the Hyperball optimizer to embeddings produced escalating gradient spikes at higher learning rates; and removing logit z-loss caused training loss to turn upward as output-normalization gains grew. The team addressed these issues by fixing the norm calculation, keeping embeddings on plain Adam, and retaining z-loss. Model sizing balanced the deadline, available compute and data, rack-level communication costs, and the need for time to train. Early comparisons favored a 360B model that fit on one rack, but improved expert parallelism made a larger 535B model feasible in a similar schedule. The design therefore reflects a deliberate tradeoff: pursue greater model capacity without sacrificing completion probability or forecastable training.