Quantization, honestly: Q4 vs Q6 vs Q8 vs FP8

Quantization is the lever that decides whether a 30B model fits in 24GB or 12GB. The tradeoff is real but smaller than the textbooks suggest — for most workloads, Q4_K_M is the right pick, and you give up less than you fear.

The bytes-per-parameter tiers

Standard GGUF conventions:

• FP16/BF16 — 2.0 bytes/param. Reference quality. A 30B model needs ~60GB.
• Q8_0 — ~1.0 bytes/param. ~30GB for 30B. Visually-lossless for most tasks.
• Q6_K — ~0.75 bytes/param. ~22.5GB for 30B. Quality recovery to within ~2 points of Q8.
• Q5_K_M — ~0.68 bytes/param. ~20.5GB for 30B. Quality/speed sweet spot for tight VRAM.
• Q4_K_M — ~0.55 bytes/param. ~16.5GB for 30B. The default for budget builds.
• Q3_K_M / Q2_K — visibly degraded. Use only when VRAM forces it.

What we actually measured

On a 30B coder model across Q4_K_M, Q6_K, and Q8_0 (see bm-003):

• Q4_K_M → Q6_K recovers ~3 points on HumanEval pass@1 while staying ~38% faster than Q8_0.
• Q8_0 barely fits 24GB and leaves no room for context.
• Q6_K is the sweet spot on 24GB Arc — quality near Q8, speed near Q4.

What quantization does NOT change

• Context length ceiling (determined by KV cache, separate budget).
• Tool-use reliability (mostly architecture, not precision).
• Licenses or safety properties of the base model.

FAQ

Does Q4_K_M noticeably hurt model quality?

Less than most people fear. On our 30B coder sweep, Q4_K_M to Q6_K recovered about 3 points on HumanEval pass@1 while staying roughly 38% faster than Q8_0. For most workloads Q4_K_M is the right default; reach for Q6_K only when you have VRAM headroom.

Which quantization should I start with?

If it fits comfortably, run Q6_K. If VRAM is tight, Q4_K_M. Use Q8_0 only when you have headroom to spare and care about edge-case quality. Avoid Q3_K_M and below unless VRAM forces it.

Does quantization change the context length I can run?

No. Context length is governed by the KV cache, which is a separate memory budget from the model weights. Quantization shrinks weights; it does not change your context ceiling.

Try it for yourself: Model Fit shows how the same model fits differently at each tier.