A tiny world model running games on a Raspberry Pi Pico 2 W

Tinyworldformer is an experiment in running a learned game simulator and game-playing agent together on a Raspberry Pi Pico 2 W microcontroller. Its central idea is a “looped” transformer: one small transformer block reuses the same weights for multiple passes, reducing memory needs. The world model takes a game frame and an action, then predicts the next frame and reward; a separate actor chooses actions from the model’s generated frames. Five small games - Breakout, Doom-lite, Flappy Bird, Pong, and Pac-Man lite - are represented as grids of tokens. On the Pico, both networks run without access to the real game, while button presses let a person take control.

The models are trained on recorded game transitions, then converted to compact int8 weights for the world model and deployed as Rust firmware. Held-out frame accuracy at two transformer passes ranges from 83.6% in Doom-lite to 100% in Flappy Bird; Doom-lite’s pixel-level score understates how often its underlying game state is correct. Tests in imagined play also expose errors that ordinary test data misses: agents can exploit small simulation flaws to achieve impossible scores, prompting additional real-game data and fine-tuning. The project details practical techniques that improve reliability, including enforcing one position for unique objects and sampling unpredictable events such as respawns. It demonstrates that a tiny learned simulator can run interactive games on inexpensive hardware, while highlighting the difficulty of keeping its imagined rules accurate under unfamiliar play.