Mini-AGI – dynamic continual learning model trained from scratch on 8GB VRAM

Mini-AGI is a small continual-learning byte-level language model designed to train from scratch on a single 8 GB VRAM GPU by storing parameters as files on disk and paging needed experts into memory. It is built so training and inference share the same forward pass: the system reads a stream of bytes, takes gradient steps on each chunk, and never relies on a frozen base. Three engineering constraints drive the design: fit within 8 GB without aggressive quantization (weights live on disk), avoid catastrophic forgetting, and accept any byte value as input (no tokenizer). The project demonstrates that a personal, continually updating model is feasible on consumer hardware, with parameter capacity bounded by free disk rather than GPU memory. Weights are not published yet; current runs are still in early passes and the project is explicitly a toy-level experiment rather than frontier-scale.

Architecturally, each character flows through two dense prelude blocks and then a single recurrent block applied up to 24 times, with a learned halting head (PonderNet) to adapt depth per token. Each block-application routes to its own top-8 experts from a shared pool via soft top-k, letting characters combine fragments from many experts; rotary positions permit extending context via continued training. Paging keeps each expert’s Adam moments with its file; a RAM cache and a VRAM working set are chosen based on recent demand with hysteresis to limit churn. New experts are added by recombination and pruned if unused. Early outputs are grammatical and on-topic but still show repetition; reading typically costs ~8 rows per character while generation costs ~9.9.