# Memorilla: 16 memory tokens let a frozen LLM read thousands of documents at once
An AI agent that remembers everything you have ever told it is the goal. The obstacle is obvious: context windows are finite, document collections grow, and feeding every relevant document into every call is expensive and slow.
Stanford's SNAP group, led by Jure Leskovec, published a solution called Memorilla. It replaces thousands of document tokens with 16 learned memory tokens, and the frozen LLM becomes more accurate on factual verification than if it had read every document in full.

How it works
Memorilla separates the act of remembering from the act of recalling. A frozen embedding encoder (Qwen3-Embedding-4B) processes each incoming document, such as a conversation or a book chapter, into an embedding that lives in a persistent store. When a question arrives, a small trainable memory module (194 million parameters, about 2% of the decoder size) reads the question and every embedding in the store. It produces 16 query-dependent vectors that the frozen Qwen3-8B decoder reads alongside the question.
The key constraints that guided the design are: every query reads the whole collection, not a pre-selected subset; what memory returns depends on the question; the token budget is fixed regardless of collection size; and new documents can be added without re-encoding the past.

The results
The paper reports results across eight benchmarks covering personalization, factual reasoning, and long-document QA. Memorilla achieved the best average score among all evaluated methods.
| Metric | Full context | Memorilla (16 tokens) | Improvement | |---|---|---|---| | FactKG accuracy | 80.43% | 84.33% | +3.9% | | Document tokens | baseline | 108x fewer | 108x reduction | | NarrativeQA compression | baseline | 4,000x | 4,000x reduction | | Latency (512 docs, H100) | 19.9s | 0.74s | 27x faster | | TextWorld agent success | 18.6% | 24.2% | +5.6% |

The four design requirements
The paper is explicit about four requirements that distinguish Memorilla from prior work. Collection-wide access means every query considers every document, not just the top-N from a retriever. Query-dependent recall means the same collection serves many different queries. The fixed token budget ensures the LLM never sees more than 16 memory tokens regardless of collection size. Reusable embeddings mean documents are encoded once and the store grows without re-encoding history.

How it compares
The practical alternative to Memorilla is retrieval-augmented generation (RAG), which picks the top-N passages and feeds them into the context. RAG is simpler, well understood, and works. Memorilla differs in that it considers every document, compresses into a fixed budget, and stays fast regardless of collection size. On the other hand, Memorilla can lose fine-grained facts that a dedicated retriever would surface, and it has only been tested with a Qwen encoder and a Qwen decoder.

What it means for agent memory
The most relevant result for agent development is the TextWorld experiment. An agent that used Memorilla to recall its own past game sessions improved its success rate from 18.6% to 24.2%. That is a 30% relative improvement from remembering what it did before, without any changes to the model or the agent architecture.
For an agent system like Argo, where agents run long-lived sessions across multiple projects and document stores, the same pattern applies: encode every completed task output into a Memorilla store, and the agent draws on the entire history through 16 tokens per query.
The limits
The paper lists its own limitations. Memorilla can lose fine-grained facts. It was tested only with a Qwen encoder and Qwen decoder. Transfer across model families is untested. It may not carry over to coding tasks. The memory module needs training for each setup, which plain retrieval does not.
The training recipe is three stages: stage 1 on Wikipedia, stage 2 on a mixture, stage 3 on multitask data. All checkpoints are on Hugging Face at memorilla/Memorilla-Qwen3-8B, Apache 2.0.
The repo
Code: github.com/snap-stanford/memorilla. Paper: arXiv 2610.21375. Weights on Hugging Face: memorilla/Memorilla-Qwen3-8B.