Memorilla: Latent Semantic Memory for LLMs

Memorilla is a memory system designed to let a frozen large language model use information from past conversations, long documents, or earlier agent actions without repeatedly placing large amounts of text in its context window. It encodes documents once into reusable embeddings, then uses each new query to retrieve and combine relevant information as a fixed set of learned “memory tokens” in the model’s input embedding space. The central idea is to make accumulated knowledge available in a compact, query-specific form, addressing the difficulty of finding and synthesizing useful details while keeping prompts short. Because the document encodings can be reused, the approach also supports settings where an agent’s history keeps growing.

In tests across personalization, long-document, and knowledge-intensive question answering, Memorilla used 16 memory tokens and outperformed comparison systems, including Mem0 and MemoRAG, on five of eight benchmarks. On FactKG, it achieved 84.33% accuracy, compared with 80.43% for inference using the full context, while representing source documents with 108 times fewer tokens. The system was also evaluated in a TextWorld environment, where an agent must act based on experience accumulated over a trajectory. Memorilla reduced repeated wandering through already explored rooms and raised average success from 18.6% with full-history in-context learning to 24.2%. The results suggest that compact learned memories can help LLM agents retain and apply information over extended interactions.