> ## Documentation Index
> Fetch the complete documentation index at: https://supermemory-vorflux-slack-workspace-override-consent.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Measuring Results

> How MemoryBench scores a run — qualitative judging and quantitative metrics

Every question goes through the same pipeline — **ingest → search → answer → evaluate → report** — and produces two different kinds of signal: a **qualitative** judgment on whether the answer was actually right, and **quantitative** measurements of how fast and how expensive getting there was. MemoryBench keeps both instead of collapsing everything into a single number.

## Qualitative: was the answer right?

Correctness isn't decided by string matching — a **judge LLM** (GPT-4o, Claude Sonnet, or Gemini Flash, your choice) compares the provider's answer against ground truth and returns a verdict:

```typescript theme={null}
{ score: 0 | 1, label: "correct" | "incorrect", explanation: string }
```

The judge is judge-agnostic on purpose — score the same run with two different judges if you want to sanity-check that a result isn't an artifact of one model's grading bias. The prompt the judge uses can also vary by question type (temporal questions get graded differently than abstention questions, for example), and providers can supply their own judge prompts if their answers need custom framing.

```bash theme={null}
# Grade a run with a different judge
bun run src/index.ts run -p supermemory -b locomo -j sonnet-4
```

## Quantitative: how fast, how much

Alongside the correctness verdict, every question also records hard numbers:

| Metric             | What it measures                                                                                                      |
| ------------------ | --------------------------------------------------------------------------------------------------------------------- |
| **Accuracy**       | Judge score (0–1) averaged across all questions, and broken down per question type                                    |
| **Latency**        | Search time and answer-generation time per question (p50 / p95 across a run)                                          |
| **Context tokens** | How much context was sent to the answering model — a proxy for retrieval cost                                         |
| **Success rate**   | Percentage of questions that completed without erroring (failures are excluded from accuracy, not counted against it) |

## MemScore

The report doesn't reduce these to one score. **MemScore** reports the three that matter for a production decision side by side:

```
MemScore: 86% / 145ms / 1823tok
          ▲     ▲       ▲
          │     │       └─ context tokens sent to the answering model (cost)
          │     └───────── search latency
          └─────────────── answer accuracy vs. ground truth
```

A provider that's 2% more accurate but 5x slower and 3x more expensive in context tokens isn't unambiguously "better" — MemScore leaves that trade-off to you instead of picking weights for you.

## Reading the numbers

Rough bands, from running MemoryBench across the built-in benchmarks:

| Accuracy | Read                                                           |
| -------- | -------------------------------------------------------------- |
| 80%+     | Excellent, production-ready                                    |
| 70–80%   | Good, some room to improve                                     |
| 60–70%   | Adequate, likely needs tuning                                  |
| `<60%`   | Investigate — retrieval, prompts, or indexing likely need work |

| Search latency | Read                            |
| -------------- | ------------------------------- |
| `<100ms`       | Excellent (vector-search level) |
| 100–300ms      | Good, typical API latency       |
| 300–500ms      | Adequate for most use cases     |
| `>500ms`       | Slow — worth optimizing         |

Per-question-type breakdowns are usually more useful than the headline number: strong on LoCoMo but weak on LongMemEval means good temporal/cross-session recall but weak dense-retrieval; the reverse means the opposite. That's what tells you what to actually go fix.

## Digging into a specific run

```bash theme={null}
bun run src/index.ts status -r my-run             # progress / summary
bun run src/index.ts show-failures -r my-run       # full context on what got graded wrong
bun run src/index.ts serve                         # web UI at localhost:3000 for visual inspection
```

Results and checkpoints for every run live at `data/runs/{runId}/report.json`, with per-question-type accuracy, latency percentiles, and per-question token counts.

***

## Next

<CardGroup cols={2}>
  <Card title="Adding a Provider" icon="plug" href="/memorybench/extend-provider">
    Get your own system into a run that produces these numbers.
  </Card>

  <Card title="Building a Benchmark" icon="flask-conical" href="/memorybench/extend-benchmark">
    Measure against scenarios specific to your product.
  </Card>
</CardGroup>