37% on ARC-AGI-2 Kaggle Without LLMs

Mithil Vakde reports scoring 37% on the ARC-AGI-2 Kaggle competition with a 100-million-parameter transformer trained from scratch, without using a large language model. The system reached sixth place and, according to Vakde, surpassed the previous year’s winning LLM. Its score rose from 33% to 35% and then 37% after looping was added near the end of development. The approach also combines all example pairs in the prompt, uses sparse hub-and-spoke attention to fit training across four L4 GPUs, and relies on custom synthetic data.

Vakde argues that looping improves parameter efficiency: at equal depth, the looped model beat non-looped stacks with as much as four times the parameters by 2.3-7.6%, and even models below one million parameters performed well. Unlike HRM and TRM, this system predicts outputs autoregressively, one pixel at a time; looping is motivated by minimum description length rather than repeatedly revising a complete answer. Vakde says hierarchical looping did not help and that the method is cheaper and faster to train. The project’s goal is to test sample efficiency: models exposed to vast datasets may already have encountered ARC puzzles, obscuring what they can learn from a small number of examples. The final model was substantially undertrained because time ran out, leaving room for improvement.