Speculative decoding on consumer GPUs: MTP measured 2.07×

Every speculative decoding explainer follows the same shape. It describes the draft-verify mechanism, calls it a "free speedup," cites a 2–3× multiplier from a paper benchmark, and moves on. That is correct as far as it goes. What it never comes with is the actual measured multiplier on consumer hardware, or the accept-rate curve across context lengths, or an honest account of when the speedup doesn't apply. The folklore says "free 2×"; the practitioner's question is "for which model, on which card, at what context, with which runtime."

This is the measured version. Qwen3.6-27B-A3B-Instruct on a single Radeon AI PRO R9700 (gfx1201 RDNA4), same sha256-verified weights, with and without the model's MTP draft head active — plus the long-context sweep that shows what happens to acceptance as the KV cache fills. The headline finding is the kind of thing only a lab can show: the multiplier is real (2.07× on this configuration), the quality cost is zero by construction, and the long-context curve is the part every abstract explainer misses.

For the runtime-layer question of which tool activates MTP and which doesn't, see Ollama vs llama.cpp on the same hardware. This article is the deep-dive on the mechanism and the measured numbers.

What speculative decoding actually does

In one paragraph. Standard autoregressive generation runs the model forward once per output token. Speculative decoding runs a small, cheap "draft" predictor forward N times to guess the next N tokens, then runs the main model forward once to verify all N guesses in parallel. Accepted guesses are "free" tokens — the main model already computed them as part of its single verify pass. Rejected guesses abort the chain at the first failure, the main model emits its own token at that position, and the cycle restarts. The speedup comes from the parallelism of verification: one forward pass of the main model checks N candidate tokens for roughly the cost of generating one. The math works because attention's cost scales with sequence length, but the verify pass's N-candidate check amortizes the per-token fixed cost.

MTP — the model-shipped version

Speculative decoding used to require a separate small "draft model" loaded alongside the main one — pick something like StableLM-1.6B as a draft for Llama-70B, eat the extra VRAM, and accept that the draft and main had to share a tokenizer and a vocabulary. Multi-Token Prediction (MTP) is the cleaner version: the model ships its own draft head as part of the release, trained alongside the main weights, with no separate model to load and no tokenizer mismatch to manage. Qwen3.6-27B-A3B-Instruct is the example in our data — the -mtp GGUF variant bundles the draft head, and a runtime that supports it activates the speculative path automatically.

The trade-off: MTP only works on models that ship the head (a small but growing list in 2025–2026), and only on runtimes that activate it (as of our measurement, llama.cpp Vulkan did; Ollama's HIP runner didn't — see bm-004).

Measured: MTP on the R9700

The cleanest speculative-vs-non-speculative comparison in our library: same single R9700, same Qwen3.6-27B-A3B-Instruct Q4_K_M-mtp weights, back-to-back runs with the MTP path toggled.

Path (same R9700, same weights)	Generation tok/s	VRAM	MTP accept rate	Mean draft length
llama.cpp Vulkan + MTP	66.0	21.9 GB	(short-prompt run)	(short-prompt run)
llama.cpp Vulkan, no MTP	31.9	18.8 GB	—	—

That is a 2.07× generation speedup from toggling one path, with the same card, the same sha256-verified weights, and the same workload. The 3.1 GB VRAM cost is the draft head's footprint. Mean draft length and accept rate for this specific short-prompt run weren't captured in bm-004 — bm-006 (below) captures them across the long-context sweep.

The honest framing: 2.07× is the headline, but it's the short-prompt, ideal-accept-rate case. The long-context picture is more nuanced.

Measured: the long-context accept-rate curve

bm-006 ran the same Qwen3.6-27B-mtp on the same R9700 across 8K to 80K context, capturing both generation speed and MTP acceptance statistics. This is the dataset every abstract explainer misses.

Target context	Generation tok/s	MTP accept rate	Mean draft length
8K	64.46	0.660	2.97
32K	61.44	0.694	3.07
56K	59.26	0.738	3.19
80K	55.70	0.752	3.26

Three findings worth pulling out.

First, generation speed holds remarkably well across context — 64→56 tok/s from 8K to 80K, only a ~1.3× decay over a 10× context expansion. MTP is doing real work here; without it, the decay would be steeper.

Second, acceptance rate actually rises as context fills (66% → 75%). Counterintuitive — you'd expect longer context to make the draft head's job harder. The mechanism: at long context, the model's predictions become more deterministic (the prompt strongly constrains the continuation), which makes the draft head's guesses more likely to match. The draft gets longer (2.97 → 3.26 mean tokens accepted per verify pass), which is what holds the generation speed up.

Third, prompt-processing speed falls (762 → 580 tok/s across the same range, not shown in the table above). This is the part that shrinks the wall-clock benefit at long context: even though per-token generation holds, you spend longer digesting the prompt before the first token. For chat workloads where TTFT matters, the speculative speedup on generation is partially offset by the prefill cost.

Does speculative decoding hurt quality?

No, by construction. The main model verifies every token the draft head proposes. Accepted tokens are exactly what the main model would have generated anyway. Rejected tokens are discarded and the main model regenerates from the rejection point. The output is bit-identical to non-speculative decoding up to floating-point noise.

This is the "free" in "free speedup" — there is no quality knob being traded. The only costs are VRAM (the draft head's footprint) and runtime compatibility (the runtime has to activate the path). The 2.07× in our data is a speed-only win.

The caveat: we did not run an empirical diff test on this model to verify bit-identity. The claim follows from the algorithm; if a runtime implements it correctly, the output is identical. If you want to verify for yourself, run the same prompt with and without MTP and diff the outputs.

Where it doesn't help

Three places speculative decoding provides no benefit:

1. Prompt processing. Speculative is generation-only. The speedup applies to emitting tokens after the prompt is digested, not to digesting the prompt in the first place. For long-prompt / short-output workloads (RAG retrieval, document summarization where the prompt is the document), most of the wall-clock is prefill and speculative doesn't touch it.
2. Models without a draft head and without a compatible separate draft. MTP requires the model to ship the head. The alternative — loading a separate small draft model — requires the draft and main to share a tokenizer, which limits pairings. If neither path is available, speculative isn't either.
3. Memory-bandwidth-bound cards. Speculative decoding's verify pass is wider but shallower than N independent forward passes. On a card where memory bandwidth already saturates at non-speculative generation, the verify pass hits the same ceiling and the speedup shrinks. Our R9700 has headroom; a more bandwidth-limited card might see less.

The honest caveats

• MTP is model-specific. The 2.07× number is for Qwen3.6-27B-A3B-mtp. Other models without a draft head will not see this speedup via MTP. Generalize carefully.
• Alternative draft approaches not measured. Medusa, EAGLE, paired-draft with a separate small model — we have no data on these on our hardware. They are real and useful; we just cannot speak to them.
• Ollama compatibility. As of bm-004, Ollama's HIP runner did not activate MTP on this model. Check current release notes; this is a moving target.
• Single-prompt workload. bm-004 and bm-006 are single-prompt, not the workload-pack-v1 suite. Don't cross-compare tok/s to bm-001/002/003.
• Accept-rate curve is one workload shape. bm-006 used one prompt type. Different workloads (code, dialogue, structured output) will produce different curves.

How to turn it on

Short, practical. Three things, in order.

1. Use the -mtp model variant. The model file must ship the draft head. For Qwen3.6-27B, that's the Q4_K_M-mtp.gguf (or equivalent) artifact, not the plain Q4_K_M.gguf.
2. Build or install a runtime that supports it. Our measured path was llama.cpp with the Vulkan backend; the MTP path was activated automatically when the model artifact contained the head. OVMS on Arc is a separate question. Check your runtime's release notes.
3. Verify it's actually active. The clearest signal is generation tok/s jumping to roughly 2× the non-MTP baseline on the same model. If you don't see the jump, the runtime is loading the model without activating the speculative path — see Ollama vs llama.cpp on the same hardware for the case where this happened.

The takeaway

Speculative decoding is the closest thing local inference has to a free speedup. On a model that ships an MTP head, on a runtime that activates it, you get ~2× generation at zero quality cost. The qualifiers matter: MTP is model-specific, runtime support is uneven, the speedup shrinks at long context relative to wall-clock time even as per-token generation holds. Pick a model that ships the head, pick a runtime that activates it, verify the speedup actually shows up — and run Model Fit to confirm the draft head's VRAM cost fits alongside the main weights.

The runtime question — Ollama vs llama.cpp and which one actually activates MTP on which backend — is in Ollama vs llama.cpp on the same hardware. The backend question — CUDA vs ROCm vs Vulkan vs OpenVINO underneath — is in the measured backend comparison. The raw records behind every number above are in the benchmark library.

Last verified: July 2026. MTP and speculative decoding support are active development areas in every major runtime — re-validate against current llama.cpp, Ollama, and OVMS release notes before relying on any specific claim about which path activates on which backend.