Training Text-to-Image Models 3.6× Faster

This describes a method to train text-to-image models far more efficiently by discarding the traditional VAE+DiT split and moving compression into the diffusion transformer itself. Latent diffusion models are bottlenecked by attention cost because VAEs only compress tokens so far (commonly to 16×16 latents), forcing huge context windows for high-resolution data. Building on the JiT idea of aggressive patchification - turning pixel patches into channel-heavy tokens - Linum introduces JiT-DDT, an encoder-decoder pixel-space DiT that recovers fine-grained details lost in naïve JiT implementations and trains much faster.

The technical insight is that the common v-prediction (velocity) objective in flow-matching diffusion fails as dimensionality rises because the model must learn high-dimensional Gaussian noise; switching to x-prediction avoids that curse of dimensionality and lets the DiT learn from high-channel inputs. JiT-DDT uses this objective plus architectural refinements: a 2.5B active pixel-space DiT with 320 pixel tokens (64 encoder + 256 decoder) versus Linum v2’s 2.0B latent DiT + VAE with 256 latents. The result is a text-to-image model that generates 512×512 images while using 3.6× fewer GPU-hours and seeing 4.2× fewer samples than the prior baseline. Code and weights are released under Apache 2.0 as a research artifact.