Memorilla: Latent Semantic Memory for LLMs

Memorilla is a memory system for frozen large language models: it condenses a collection of documents - such as chat histories, books, or game experiences - into 16 query-dependent memory tokens, then gives those tokens to the model alongside a question. The fixed memory budget is intended to make long-term information usable without filling the model’s context window. In a personalization example, it retrieves a family-health detail from one of 107 past conversations to shape an answer to an unrelated breakfast question. The repository provides code, pretrained checkpoints, training and evaluation recipes, and instructions for using custom question-and-answer datasets.

Reported results indicate that this compression can preserve useful information while reducing prompt size and inference costs. On FactKG, Memorilla reaches 84.33% accuracy, compared with 80.43% when the model receives every document, while using 108 times fewer tokens. It compresses NarrativeQA books and scripts by more than 4,000 times; with up to 512 documents, response time stays at 0.74 seconds, whereas full-context prompting rises to 19.9 seconds. The system also improves average success on TextWorld games from 18.6% to 24.2% when agents learn from their histories using reward-based training. Running the supplied training recipes requires substantial hardware: the three main stages take about 19 hours on four H100 GPUs, and installation targets Linux, CUDA, and Python 3.11.