Skip to main content
Benchmarking memory systems is hard, and most comparisons you’ll find aren’t apples-to-apples — different datasets, different judges, different prompts, cherry-picked runs. MemoryBench (supermemoryai/memorybench, MIT licensed) is the open-source framework we built to fix that: the same benchmark questions, the same pipeline, and the same judges run against every provider, so a comparison actually means something. We open-sourced it so you don’t have to take our word for anything — you can run it yourself, against your own memory implementation, on the datasets that match your use case.

git.new/membench

Clone it, run it against your own data, or read the source for exactly how each provider is scored.

​
Benchmark your own memory system

MemoryBench ships a Claude Code skill that automates the entire process of benchmarking a custom memory implementation — yours — against Supermemory, Mem0, and Zep. Point it at your code and it handles discovery, integration, and the run:
  1. Asks a few questions about your setup (provider name, where your memory code lives, which benchmark, which competitors, how many questions)
  2. Analyzes your memory code to find its init, ingest, and search methods
  3. Generates a provider adapter and registers it with the framework
  4. Runs the full benchmark against your chosen competitors
  5. Reports accuracy, latency, and context-token results side by side
No manual TypeScript required to get a first result — see Adding a Provider for what the skill generates and how to do it by hand.

​
How it works

Every run goes through the same checkpointed pipeline, regardless of provider or benchmark:
Ingestion can take hours for large datasets, and API calls fail — so every phase checkpoints independently and a run resumes from the last completed step instead of starting over.

Supermemory

Chunk-based semantic search

Mem0

LLM-powered memory extraction

Zep

Knowledge graph construction
None of these fit your use case? Build your own benchmark — providers, benchmarks, and judges are all pluggable. Judging is judge-agnostic too — score a run with GPT-4o, Claude, Gemini, or any model you configure, so results aren’t an artifact of one evaluator’s bias.

Building a Benchmark

Test on scenarios that actually match your product, not just the built-in datasets.

Adding a Provider

Register your own memory system so it can be scored and compared.

Measuring Results

MemScore, judge scoring, and how to read a report.

Report an issue

Found a bug, or want a provider/benchmark we don’t support yet?