Quantization in 2026: Q4_K_M is no longer the compromise

If there's one piece of local-AI folklore that needs retiring, it's "Q4 is broken, use Q8." That was defensible in 2024. It isn't anymore. The Q4_K_M quantization tier — the practical default for almost every model we recommend — produces quality within a few points of Q8 on real workloads, at roughly half the VRAM. The "compromise" framing is two years out of date.

We can show this with our own data rather than ask you to take it on faith.

What quantization actually does

A model's weights start as 16-bit floats (FP16/BF16). Quantization packs those weights into fewer bits — 8-bit (Q8), 6-bit (Q6_K), 5-bit (Q5_K_M), 4-bit (Q4_K_M), down to 2-bit (Q2_K). Fewer bits per parameter means:

• Smaller file — a 30B model goes from ~60GB at FP16 to ~18GB at Q4_K_M.
• Less VRAM — you can fit a much bigger model on the same card.
• Some quality loss — this is the part that used to matter a lot and now matters a little.

The "K" variants (K-quants, from the GGUF/llama.cpp lineage) are smarter than uniform quantization: they allocate precision where it matters most (important weight matrices get higher precision). The "_M" (medium) is the balanced mix. Hence Q4_K_M.

The measured delta, on Qwen3-Coder-30B

In bm-003 we ran the same model — Qwen3-Coder-30B-A3B — at Q4_K_M, Q6_K, and Q8_0 on the same hardware, same workload, same everything else. The point was to isolate just the quantization variable.

Quant	~Footprint	HumanEval pass@1	Quality vs Q8_0	VRAM headroom on 24GB
Q8_0	~32GB	(baseline)	—	doesn't fit
Q6_K	~24GB	−1 to −2 pts	~negligible	~0GB
Q4_K_M	~18GB	−3 pts	small	~6GB

Read it honestly: Q4_K_M is within ~3 points of Q8_0 on HumanEval pass@1, at a bit over half the footprint. Q6_K is essentially indistinguishable from Q8 on quality. The thing that used to be true — "Q4 quality falls off a cliff" — is not what the 2026 numbers show.

(The full record — every flag, every score, every caveat — is in bm-003. We're summarizing; the data is the source of truth.)

Why this changed

Two things moved between 2024 and 2026, and both matter:

1. K-quants got good. The naive "round every weight to 4 bits" approach did break quality. The K-quant method (and the newer importance-aware schemes) preserve the weights that matter and aggressively quantize the ones that don't. The result is that "4-bit" in 2026 is not the same animal as "4-bit" in 2024.
2. The models got more robust. Modern training pipelines expose models to lower precision during training, so the deployed weights tolerate quantization better. A 2026 30B model quantizes more gracefully than a 2024 7B did.

None of this is magic — there is a quality delta at Q4. It's just small enough now that for almost every local workload, the VRAM you save is worth more than the points you lose.

When you actually should step up

Q4_K_M is the default, not a religion. Step up a tier when:

• You're doing hard reasoning and every point matters. If you're using the model for novel algorithm design or competition math, Q6_K's near-zero quality delta is worth the VRAM if you have it.
• You have VRAM to burn and the model fits at Q6/Q8 anyway. On a 24GB card running a 7B, there's no reason not to run Q8 — the model fits with room to spare, take the quality.
• You're benchmarking and need the reference. Q8_0 is the closest-to-original you can run on consumer hardware; use it as your baseline, then drop to Q4 for daily use.

Conversely, step down to Q4_K_M (or smaller) when:

• The model doesn't fit at Q8 on your card. This is most people, most of the time. A 30B at Q8 needs ~32GB; at Q4 it needs ~18GB. That's the difference between "buy a second card" and "it just works."
• You're running an agent loop and tok/s matters more than the last quality point. Lower-precision weights compute faster. The speed delta isn't huge, but it's real, and at agent-loop scale it compounds.

The honest caveats

• One model isn't every model. bm-003 is Qwen3-Coder-30B-A3B. Smaller models (7B and below) sometimes show larger relative quality deltas at Q4 because there's less redundancy to absorb the precision loss. The direction holds; the exact points don't generalize universally.
• HumanEval isn't every task. Coding pass@1 is a reasonable proxy for "does it still work," but long-context recall, structured output, and multilingual performance can quantize differently. We report what we measured.
• Q2_K and Q3_K_M are still meaningfully worse. The "Q4 is fine now" claim does not extend down to Q2/Q3. Those tiers exist for emergencies (you absolutely must fit a 70B on a 24GB card), not for daily use. The quality cliff is real down there.

What this means in practice

For almost everyone reading this: start at Q4_K_M, step up only if you have a specific reason. That's the entire shift. In 2024 the default advice was "Q8 unless you can't"; in 2026 it's "Q4_K_M unless you have a reason not to." The reasons still exist — they're just rarer than the folklore suggests.

When you read a benchmark or a model review anywhere, the first thing to check is what quant it was run at. A number reported without a quant tier is a number reported without its most important context. Ours always have it; we'd encourage you to hold everything else to the same bar.

The full sweep — Q4_K_M, Q6_K, Q8_0 on the same model, same hardware, same workload — is in bm-003. And if you want to know whether your target model at your target quant fits on your card, Model Fit does the VRAM math.