Parallel Language Models via Variational Decoding

Parallel Language Models (PLMs) propose a method for generating entire token sequences in one forward pass rather than emitting a single token per step. They route dependency structure through learned latent auxiliary variables and apply a variational decoding objective that decomposes into a standard autoregressive loss plus an explicit parallel-decoding loss. The variational framework enforces mutual consistency among simultaneously generated tokens, enabling a single forward computation to capture the conditional dependencies normally handled autoregressively.

Empirical results place PLMs on a superior speed-quality Pareto frontier compared to autoregressive and discrete-diffusion baselines: pretrained PLMs reach near-autoregressive pretraining loss across model scales from 80M to 3B parameters, and inference speedup scales with model size roughly as a power law. After post-training, a 1.3B-parameter PLM matches or exceeds the performance of an identically trained autoregressive model on math, coding, and instruction-following benchmarks while delivering 2.3-5.7× faster inference. The work demonstrates that parallel generation can be learned from a single training objective without sacrificing autoregressive-level quality, and code is planned to be released.