Lowvram: A CLI tool to run large AI models in low VRAM GPUs

Lowvram is a Linux command-line tool for attempting to run CUDA applications whose memory requirements exceed a GPU’s physical VRAM, without modifying the applications themselves. It was developed after YuE2, a music-generation model requiring 24 GB of VRAM, proved incompatible with a system using a 6 GB RTX 3060. Rather than patching YuE2’s code for CPU offloading, the tool intercepts CUDA memory-allocation calls and substitutes managed-memory allocation. The NVIDIA driver can then keep active data in VRAM and move less-used pages to system RAM as needed. A wrapper or LD_PRELOAD activates the hooks before the target program starts, while the application continues to use ordinary device pointers.

The approach prioritizes whether a workload can run over peak speed. In the reported YuE2 test, lowvram achieved 43.1 tokens per second, close to a 45.3-token-per-second reference run using only the autoregressive portion of the model; standard execution could not load the model, and Accelerate managed 1.9 tokens per second. The tool also allowed Llama-TTS to run a Qwen3-TTS model that otherwise failed with an out-of-memory error. These results depend on moving data over PCIe, so performance can suffer when the working set exceeds VRAM. Managed allocations may also consume substantial system RAM, and the Linux-specific setup requires familiarity with CUDA.