NCM: Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation

Next-Concept Modeling (NCM) is a text-generation framework designed to combine the speed of parallel generation with the quality of more gradual decoding. It addresses weaknesses in diffusion language models: quality can fall sharply when sampling steps are reduced, each step processes the whole sequence, and training may require many dataset passes. NCM first uses a multiscale residual vector-quantized variational autoencoder to represent text as a hierarchy of discrete codes. A transformer then generates that hierarchy from coarse to fine, predicting all codes at each level in parallel. This staged design aims to capture broad structure before filling in detail, while requiring fewer sequential generation stages.

Across the reported evaluations, NCM generally improves the speed-quality tradeoff. At comparable parameter scales on LM1B, it achieves better generative perplexity than the strongest diffusion baseline while running 21 times faster. It is competitive with that baseline on OpenWebText at 10 times the speed, and matches its ROUGE-L score on XSum while running 31 times faster. On Sudoku, a reasoning task, NCM outperforms both autoregressive and diffusion baselines across difficulty levels while using fewer function evaluations. Together, these results suggest that hierarchical representations and parallel prediction within each level can support efficient generation without sacrificing benchmark performance.