Qwen3.6-27B + MTP at long context: where speculative decoding stops paying off (Radeon AI PRO R9700)
MTP speculative decoding on Qwen3.6-27B (single Radeon AI PRO R9700, llama.cpp Vulkan) holds 55-65 tok/s generation across 8K-80K context - degrading roughly 1.3x over that range - while prompt processing falls from ~760 to ~580 tok/s as context fills. The practical finding: MTP keeps working at long context (acceptance 66-75%), but KV-cache pressure steadily erodes both prompt and generation speed. Founder's prior short-prompt runs hit ~75-86 tok/s; the numbers here use real document prompts and are correspondingly a bit lower.
Configuration
- Model
- Qwen3.6-27B-A3B-Instruct (MTP-enabled GGUF)
- Artifact
- Qwen3.6-27B-Q4_K_M-mtp.gguf
- Checksum
- sha256:pending-capture-on-next-load
- Quantization
- q4-k-m
- Context length
- 98,304 tokens
- Backend
- vulkan
- Runtime
- llama.cpp Vulkan build (r9700-final), MTP speculative decoding (--spec-type draft-mtp --spec-draft-n-max 3)
- Settings
- -m Qwen3.6-27B-Q4_K_M-mtp.gguf -dev Vulkan1 (SINGLE R9700, no tensor split) -ngl 999 (FULL offload - critical) -c 98304 -b 2048 -ub 512 -ctk f16 -ctv f16 -fa on --spec-type draft-mtp --spec-draft-n-max 3 --parallel 1
- Workload pack
- longctx-decay-sweep-v2
- Author
- Edgar
- Reviewer
- Edgar
Results
Context-length sweep (8K, 32K, 56K, 80K targets) against a standardized reproducible technical document (varied 6-section prose about benchmark methodology - NOT repetitive synthetic text, which was found to skew MTP acceptance). Each request: real document prompt + 'In one sentence, what is the methodology about?' + 256-token generation. Server reports pure prompt-processing and eval (generation) rates separately (the authoritative numbers). NOT the site's workload-pack-v1 (100-prompt HumanEval) - this is a long-context throughput/decay characterization.
64.5tok/s
GEN · Ray
763tok/s
PROMPT
0ms
TTFT
0.0GB
VRAM USED
| MACHINE | PROMPT tok/s | GEN tok/s | TTFT ms | VRAM GB | RAM GB | POWER W |
|---|---|---|---|---|---|---|
Ray 8K-context run (task 0). prompt_tokens=7,302. MTP draft acceptance 0.660 (mean length 2.97). Single R9700 (Vulkan1), -ngl 999. Server-reported eval rate = 64.46 tok/s over 256 generated tokens. | 762.8 | 64.5 | 0 | 0.0 | 0.0 | 0 |
Ray 32K-context run (task 93). prompt_tokens=22,583. MTP draft acceptance 0.694 (mean length 3.07). Generation 4.7% slower than 8K. | 836.1 | 61.4 | 0 | 0.0 | 0.0 | 0 |
Ray 56K-context run (task 190). prompt_tokens=22,573 (note: the 56K-target doc tokenized to ~22.5K - char/token estimate was rough). MTP acceptance 0.738 (mean length 3.19). Generation 8.0% slower than 8K. | 685.1 | 59.3 | 0 | 0.0 | 0.0 | 0 |
Ray 80K-context run (task 284). prompt_tokens=22,823 (same tokenization caveat). MTP acceptance 0.752 (mean length 3.26). Generation 13.6% slower than 8K - the decay curve is the headline finding. | 579.5 | 55.7 | 0 | 0.0 | 0.0 | 0 |
Limitations
- ◆Tokenization mismatch: the 32K/56K/80K TARGET token counts produced ~22.5K ACTUAL prompt_tokens (the char/token estimate of ~4 chars/token was rough for this content). The decay trend is still valid (8K actual=7,302 -> 80K-target=22,823 actual), but the context sizes aren't exactly the labeled values. A follow-up using the tokenizer's exact count would tighten this.
- ◆ttft_ms, vram_used_gb, ram_used_gb, power_watts are 0 - not captured by the llama-server log in this run.
- ◆Founder's prior short-prompt / high-MTP-acceptance runs hit ~75-86 tok/s; the numbers here (55-65 tok/s) use real document prompts which depress MTP acceptance slightly. Both are valid; they measure different things (peak short-prompt vs realistic-context decode).
- ◆Workload is a characterization sweep, not workload-pack-v1. Do not cross-compare generation tok/s against workload-pack-v1 results.
- ◆Single-GPU config (one R9700, no tensor split) was chosen because dual-GPU testing showed MTP underperforms with cross-GPU sync overhead on small decode batches.
- ◆Qwen3.6 emits a <think> reasoning block; with max_tokens=256 and a one-sentence ask, thinking consumed most of the budget on some runs (the 256 generated tokens include any thinking).
- ◆model_checksum not captured. Raw evidence (/tmp/mtp_recipe.log on Ray) to be uploaded to /lab/raw/bm-006/ before or within 30 days of launch (AC-007).
Corrections
2026-07-18
METHODOLOGY CORRECTION (same day as initial publication). The first version of bm-006 used -ngl 99 (and -c 262144), which under-offloaded layers to the GPU and produced numbers ~2.5-3x too slow (28.7 tok/s @ 8K, 20.4 @ 65K). Re-run with the founder's documented winning recipe (-ngl 999 full offload, -c 98304, otherwise identical) yields 64.5 tok/s @ 8K and 55.7 @ 80K - matching the founder's independently-recovered memory of ~60-86 tok/s on this exact hardware. The earlier wrong numbers have been replaced in this record; the lesson (-ngl must be high enough to fully offload all layers) is captured for future runs.