How we test

This is the contract behind every benchmark on the site. If a report doesn't follow this protocol, it shouldn't be here.

1 · The principle

Every published number is reproducible from the published configuration. A reader with the same hardware and software stack should be able to recreate our result within the stated variance. If they can't, we publish a correction.

2 · What we record

Every benchmark captures, at minimum:

• Model artifact — file name, source, and SHA-256 checksum when local.
• Quantization — exact GGUF tier (e.g. Q4_K_M, not "4-bit").
• Runtime + version — llama.cpp build hash, vLLM version, etc.
• Runtime settings — every relevant flag (-ngl, -c, -t, -b, --no-mmap).
• Hardware — exact machine from the registry (CPU, RAM, GPU, VRAM, OS).
• Backend — CUDA, ROCm, Vulkan, SYCL, Metal, or CPU. Driver/runtime version pinned.
• Context length — tokens, with KV cache size noted.
• Workload — which prompt pack was used and how it was scored.
• Date — ISO date the test was run.
• Author + reviewer — who ran it and who checked it.

3 · Measurements

For each machine in a benchmark, we record:

• Prompt processing speed — tokens/second during prompt ingestion.
• Generation speed — tokens/second during output. This is the headline number.
• Time to first token (TTFT) — milliseconds from request to first output token.
• VRAM used — gigabytes of GPU memory at steady state.
• RAM used — gigabytes of system memory.
• Power (where available) — watts at the wall during sustained generation.

4 · Run protocol

Every reported number is the average of 5 measured runs preceded by 3 warmup runs that are discarded. Run-to-run variance is reported in the limitations section; if variance exceeds 10% on the headline metric, we say so explicitly and investigate before publishing.

5 · Workload pack v1

The first workload pack covers five categories:

• Coding — 100 HumanEval-style prompts, scored pass@1.
• General — 50 mixed reasoning/writing prompts, scored blind against a reference.
• Long context — 8K–32K-token prompts, scored on factual recall.
• Structured output — prompts requiring valid JSON or function calls, scored on schema validity.
• Real app — at least one task drawn from a shipping local-first product (currently MoreYess's sales-call rehearsal workload).

6 · Backend policy

We test on the backend that makes sense for the hardware — but we always say which. One specific rule: Intel Arc Battlemage uses Vulkan, not SYCL. SYCL produces garbage output on Battlemage (BMG G21); this is a known issue and we will not publish SYCL numbers for Arc B-series cards.

7 · Evidence retention

Raw outputs (logs, JSON timing data, screenshots) are stored durably and linked from each report via the raw data mark. Records are immutable after publication; corrections create a new revision with a visible change log.

8 · Corrections

If we're wrong, we fix it openly. Every correction is dated, summarized on the report, and listed on /corrections. We never silently edit a published number. Our guardrail is fewer than 2% of records requiring a correction caused by preventable data errors.

9 · What we don't do

• Rank products by commission or commercial interest.
• Publish numbers we can't reproduce.
• Hide settings that would change the result.
• Accept payment for placement in benchmarks.
• Use unverified community-submitted numbers as if they were lab-verified.