If there’s one piece of local-AI folklore that needs retiring, it’s “Q4 is broken, use Q8.” That was defensible in 2024. It isn’t anymore. The Q4_K_M quantization tier — the practical default for almost every model we recommend — produces quality within a few points of Q8 on real workloads, at roughly half the VRAM. The “compromise” framing is two years out of date.
We can show this with our own data rather than ask you to take it on faith.
What quantization actually does
A model’s weights start as 16-bit floats (FP16/BF16). Quantization packs those weights into fewer bits — 8-bit (Q8), 6-bit (Q6_K), 5-bit (Q5_K_M), 4-bit (Q4_K_M), down to 2-bit (Q2_K). Fewer bits per parameter means:
- Smaller file — a 30B model goes from ~60GB at FP16 to ~18GB at Q4_K_M.
- Less VRAM — you can fit a much bigger model on the same card.
- Some quality loss — this is the part that used to matter a lot and now matters a little.
The “K” variants (K-quants, from the GGUF/llama.cpp lineage) are smarter than uniform quantization: they allocate precision where it matters most (important weight matrices get higher precision). The “_M” (medium) is the balanced mix. Hence Q4_K_M.
The measured delta, on Qwen3-Coder-30B
In bm-003 we ran the same model — Qwen3-Coder-30B-A3B — at Q4_K_M, Q6_K, and Q8_0 on the same hardware, same workload, same everything else. The point was to isolate just the quantization variable.
| Quant | ~Footprint | HumanEval pass@1 | Quality vs Q8_0 | VRAM headroom on 24GB |
|---|---|---|---|---|
| Q8_0 | ~32GB | (baseline) | — | doesn’t fit |
| Q6_K | ~24GB | −1 to −2 pts | ~negligible | ~0GB |
| Q4_K_M | ~18GB | −3 pts | small | ~6GB |
Read it honestly: Q4_K_M is within ~3 points of Q8_0 on HumanEval pass@1, at a bit over half the footprint. Q6_K is essentially indistinguishable from Q8 on quality. The thing that used to be true — “Q4 quality falls off a cliff” — is not what the 2026 numbers show.
(The full record — every flag, every score, every caveat — is in bm-003. We’re summarizing; the data is the source of truth.)
Which quant should you run? (decision table)
This is the decision table for the one configuration we measured: Qwen3-Coder-30B-A3B on Jitori’s Arc B60 Pro, Vulkan, and the identical workload-pack-v1 workload in bm-003. Tok/s is results[].generation_speed_tps; the quality delta is calculated from quality_scores[].score against Q8_0’s 0.86. The record did not include Q5_K_M, so it gets no invented midpoint.
| Quant | Measured generation | Quality vs Q8_0 | Pick this if… |
|---|---|---|---|
| Q4_K_M | 38.6 tok/s | -5 HumanEval pass@1 points | you want the fastest measured option and the most VRAM headroom; this run used 18.1GB. |
| Q5_K_M | Not measured | Not measured | you specifically want to test a middle tier yourself; bm-003 has no Q5 result. |
| Q6_K | 27.4 tok/s | -2 HumanEval pass@1 points | you have nearly a full 24GB available and want the closest measured quality result below Q8; this run used 22.6GB. |
| Q8_0 | 19.8 tok/s | baseline (0.86) | you need the reference tier and have the capacity; this run saturated 24.0GB. |
The original prose’s “~3 points” summary for Q4 was too optimistic. The benchmark record’s scores are 0.81 for Q4_K_M and 0.86 for Q8_0, a five-point difference; the table follows the record rather than revising its historical measurement. bm-003
What has changed since publication
September 13, 2026 - Mesa can change the absolute throughput picture on Battlemage
A third-party llama.cpp report found that changing only Mesa from 26.0.8 to 26.1.7 on an Arc B580 roughly doubled Q4_K_M decode throughput for its two tested models (30.25 to 66.99 tok/s and 29.79 to 61.01 tok/s). The report is not a re-run of this Qwen3-Coder-30B sweep: it used a different GPU, models, runtime path, and workload. It therefore does not replace the Q4/Q6/Q8 numbers or quality results in bm-003, but it is a reason to treat this article’s absolute tok/s as a dated record and re-test your own stack after a driver change.
Why this changed
Two things moved between 2024 and 2026, and both matter:
- K-quants got good. The naive “round every weight to 4 bits” approach did break quality. The K-quant method (and the newer importance-aware schemes) preserve the weights that matter and aggressively quantize the ones that don’t. The result is that “4-bit” in 2026 is not the same animal as “4-bit” in 2024.
- The models got more robust. Modern training pipelines expose models to lower precision during training, so the deployed weights tolerate quantization better. A 2026 30B model quantizes more gracefully than a 2024 7B did.
None of this is magic — there is a quality delta at Q4. It’s just small enough now that for almost every local workload, the VRAM you save is worth more than the points you lose.
When you actually should step up
Q4_K_M is the default, not a religion. Step up a tier when:
- You’re doing hard reasoning and every point matters. If you’re using the model for novel algorithm design or competition math, Q6_K’s near-zero quality delta is worth the VRAM if you have it.
- You have VRAM to burn and the model fits at Q6/Q8 anyway. On a 24GB card running a 7B, there’s no reason not to run Q8 — the model fits with room to spare, take the quality.
- You’re benchmarking and need the reference. Q8_0 is the closest-to-original you can run on consumer hardware; use it as your baseline, then drop to Q4 for daily use.
Conversely, step down to Q4_K_M (or smaller) when:
- The model doesn’t fit at Q8 on your card. This is most people, most of the time. A 30B at Q8 needs ~32GB; at Q4 it needs ~18GB. That’s the difference between “buy a second card” and “it just works.”
- You’re running an agent loop and tok/s matters more than the last quality point. Lower-precision weights compute faster. The speed delta isn’t huge, but it’s real, and at agent-loop scale it compounds.
The honest caveats
- One model isn’t every model. bm-003 is Qwen3-Coder-30B-A3B. Smaller models (7B and below) sometimes show larger relative quality deltas at Q4 because there’s less redundancy to absorb the precision loss. The direction holds; the exact points don’t generalize universally.
- HumanEval isn’t every task. Coding pass@1 is a reasonable proxy for “does it still work,” but long-context recall, structured output, and multilingual performance can quantize differently. We report what we measured.
- Q2_K and Q3_K_M are still meaningfully worse. The “Q4 is fine now” claim does not extend down to Q2/Q3. Those tiers exist for emergencies (you absolutely must fit a 70B on a 24GB card), not for daily use. The quality cliff is real down there.
FAQ
How much speed do I lose going from Q4 to Q8?
On this measured configuration, generation fell from 38.6 tok/s at Q4_K_M to 19.8 tok/s at Q8_0: 18.8 tok/s, or about 49% slower relative to Q4. That is a single-model, single-machine result, not a universal conversion rate. bm-003
Is Q4 good enough for coding?
For this Qwen3-Coder-30B run, Q4_K_M scored 0.81 HumanEval pass@1 versus Q8_0’s 0.86 while generating faster and using less VRAM. That makes it a defensible coding default when fit and speed matter, with the record’s warning that its single-pass quality deltas are directional rather than definitive. bm-003
How do I do the VRAM math?
Use Model Fit with your model, quantization tier, card, and intended context length. The sweep shows why: its Q4_K_M, Q6_K, and Q8_0 runs used 18.1GB, 22.6GB, and 24.0GB respectively at 8K context, so weights alone are not the whole fit decision. bm-003
Do quants affect different model sizes differently?
This site does not yet have a matched Q4/Q5/Q6/Q8 quality sweep across multiple model sizes, so its benchmark record cannot support a quantified cross-size rule. Treat this Qwen3-Coder-30B result as evidence for this model and configuration, then test the target model rather than importing its five-point Q4-to-Q8 delta. bm-003
What this means in practice
For almost everyone reading this: start at Q4_K_M, step up only if you have a specific reason. That’s the entire shift. In 2024 the default advice was “Q8 unless you can’t”; in 2026 it’s “Q4_K_M unless you have a reason not to.” The reasons still exist — they’re just rarer than the folklore suggests.
When you read a benchmark or a model review anywhere, the first thing to check is what quant it was run at. A number reported without a quant tier is a number reported without its most important context. Ours always have it; we’d encourage you to hold everything else to the same bar.
The full sweep — Q4_K_M, Q6_K, Q8_0 on the same model, same hardware, same workload — is in bm-003. And if you want to know whether your target model at your target quant fits on your card, Model Fit does the VRAM math.
The same quantization philosophy applies to image and video models — GGUF Q8 on Flux gives you ~99% of FP16 quality at half the VRAM, just like Q4_K_M gives you most of Q8 quality at half the size on LLMs. For the full deep-dive on the ComfyUI optimization stack (Dynamic VRAM, the FP16/FP8/GGUF comparison, and attention backend selection), see ComfyUI optimization in 2026.