After 600 experiments, here's what I discovered about JEPA world models

Haddy Alchaer reports 600 experiments using LeWM, a JEPA world model for the Push-T task, where a robot pushes a block to a target. Rather than predict pixels, the system represents scenes as embeddings, predicts how actions change them, and searches for action sequences that reach a goal. Replacing its learned predictor with perfect simulator physics barely changed long-range performance: success was 78% versus 77% on tasks 75 steps away. The larger limitation was planning horizon. LeWM performed well over 25 steps but achieved only 22% when planning 75 steps at once; breaking the task into shorter segments raised success to 80%. A graph of states from training demonstrations supplied those subgoals, and charging for jumps between demonstrations prevented implausible routes, bringing success to 85%.

The standard benchmark often asks for goals already represented in demonstrations, so Alchaer created harder tests that combine starts and goals from different demonstrations. A memory graph reached 45% on these tasks; adding shifted, rotated, or mirrored demonstration segments raised that to 56%, especially for goals poorly covered by the original data. A separate flaw came from scoring the entire goal image: because it includes the pusher’s position, the planner could move the block away from its target to match where the pusher appeared. With exact physics, scoring only the block increased success from 9 of 50 tasks to 39 of 50. The experiments argue that better planning, memory coverage, and goal definitions can matter far more than improving prediction accuracy, enabling long-horizon and unfamiliar goals without retraining.