Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data

This work proposes an architecture that learns from live user interaction by generating model weights on the fly rather than relying solely on a fixed parameter bank or transient prompts. Drawing inspiration from Mixture-of-Experts, a compact hypernetwork maps runtime data into a low-rank modulation of a shared base network so that feed‑forward weights are compiled from current context. Crucially, the generator maintains a Bayesian posterior over a latent code and updates it online as a session proceeds, so generated weights evolve with incoming corrections and facts instead of being fixed after a single read. The stored model footprint remains constant while the space of possible compiled weights is effectively unbounded.

That design is argued to carry runtime knowledge in weights rather than prompts, which amortizes compute across tokens, frees context window capacity, persists information across turns, and can generalize beyond in‑context retrieval. The paper specifies an evaluation protocol to compare this “infinite‑parameter” approach against in‑context learning and retrieval baselines. Key specifics are the hypernetwork→low‑rank weight modulation, online Bayesian latent updates, and tests that measure persistence, compute efficiency, and generalization of behavior supplied at run time.