Next Concept Models

Diffusion language models offer parallel sampling but suffer quality loss with few sampling steps, repeatedly process entire sequences, and need many dataset passes to reach strong performance. Next-Concept Modeling (NCM) is a two-stage, coarse-to-fine approach that replaces per-step full-sequence processing with a multiscale residual VQ-VAE that encodes text into a hierarchy of discrete codes, and a transformer that generates one level at a time, predicting all codes within each level in parallel. By operating on discrete, hierarchical representations, NCM retains parallelism while narrowing the generation task at each scale, enabling faster sampling and more efficient learning at matched parameter budgets.

Empirically, NCM achieves substantially faster inference with competitive or better quality: on LM1B it attains superior generative perplexity versus a strong diffusion baseline while running 21× faster; on OpenWebText it matches top diffusion performance at 10× speedup; on XSum it equals peak ROUGE-L while being 31× faster. NCM also excels at structured reasoning, outperforming both autoregressive and diffusion baselines on Sudoku across all difficulty levels using fewer function evaluations. The combination of multiscale VQ-VAE compression and level-wise parallel decoding provides a practical path to coarse-to-fine, high-throughput text generation.