FreeLLMStanford releases Memorilla: 16-token memory module for frozen LLMs
Memorilla compresses whole document collections into 16 tokens, boosting accuracy on benchmarks while cutting token usage by over 100×.
Stanford researchers have open‑sourced Memorilla, a latent memory system that encodes an entire document collection into a fixed set of 16 memory tokens for a frozen LLM.
The module uses a 194‑million‑parameter trainable memory network paired with a frozen Qwen3‑Embedding‑4B encoder and Qwen3‑8B decoder, running on Linux with CUDA GPUs and Python 3.11.
On the FactKG benchmark, a frozen Qwen3‑8B with Memorilla achieved 84.33% accuracy, surpassing the 80.43% score when the model sees every document and using 108× fewer tokens.
Across eight benchmarks Memorilla averaged 49.68 points, the highest reported score, and improved a TextWorld agent’s success rate from 18.6% to 24.2% with just eight memory tokens.
Training required about 19 hours on four NVIDIA H100 80 GB GPUs; inference needs a single 80 GB GPU, with response times of 0.74 s for 512 documents versus 19.9 s for full‑context prompting.
What it means for you
Memorilla lets developers keep frozen, high‑performing LLMs while dramatically reducing context length, making retrieval‑augmented applications faster and cheaper. For users of free or locally hosted models, the approach offers a way to handle massive document stores without scaling hardware or token costs.
Sources
- Stanford Research Open-Sources Memorilla, a Latent Memory Module for Frozen LLMs, MarkTechPost, 14 Nov 2026
- GitHub - snap-stanford/memorilla: 🦍 Memorilla: Latent Semantic Memory for LLMs · GitHub, github.com (official), 14 Nov 2026
Written automatically from the sources above, then checked by our code: every number and name must appear in them, and no sentence may copy a source. Spotted a mistake? Report it.
Which AI can you use for free today?
Every free AI model, ranked by blind human votes, with the memory and speed it needs on your own GPU.