Baidu dropped ERNIE 5.5 Flash Base

ERNIE 5.5 Flash Base is an openly released language-model checkpoint intended as a starting point for developers’ own domain adaptation, instruction tuning, reasoning research and agent training. It contains 280 billion parameters, activates 14 billion per token, and was trained from scratch on 33.5 trillion tokens. The checkpoint is released immediately after pre-training, with no mid-training or post-training, making it a baseline for testing what later training methods contribute. Its training context reached 128K tokens; fine-tuning and inference support contexts up to one million tokens. Baidu presents efficiency as a central goal, particularly for long-context use and repeated sampling during further training.

The 43-layer model combines sparse attention with a mixture-of-experts design. Most layers compress history into entries representing 128 tokens, while a smaller number retain token-level access and use sparse selection to control long-context compute. Its MoE layers use multiple routers to select among 512 experts, alongside a shared expert; latent-space processing reduces computation and communication. Other design choices include four parallel residual streams, a learnable output head intended to preserve plausible next-token choices while suppressing unlikely ones, and ECHO, which reuses the final layer to predict future tokens and improve the backbone’s learning. A new 200,000-entry tokenizer groups digits from right to left to support number-heavy text. The training used Muon optimization and FP8 mixed precision, with a broad multilingual corpus emphasizing capabilities such as repository-scale code and long-context understanding.