Quantization, honestly: Q4 vs Q6 vs Q8 vs FP8

Quantization is the lever that decides whether a 30B model fits in 24GB or 12GB. The tradeoff is real but smaller than the textbooks suggest — for most workloads, Q4_K_M is the right pick, and you give up less than you fear.

The bytes-per-parameter tiers

Standard GGUF conventions:

• FP16/BF16 — 2.0 bytes/param. Reference quality. A 30B model needs ~60GB.
• Q8_0 — ~1.0 bytes/param. ~30GB for 30B. Visually-lossless for most tasks.
• Q6_K — ~0.75 bytes/param. ~22.5GB for 30B. Quality recovery to within ~2 points of Q8.
• Q5_K_M — ~0.68 bytes/param. ~20.5GB for 30B. Quality/speed sweet spot for tight VRAM.
• Q4_K_M — ~0.55 bytes/param. ~16.5GB for 30B. The default for budget builds.
• Q3_K_M / Q2_K — visibly degraded. Use only when VRAM forces it.

What we actually measured

On a 30B coder model across Q4_K_M, Q6_K, and Q8_0 (see bm-003):

• Q4_K_M → Q6_K recovers ~3 points on HumanEval pass@1 while staying ~38% faster than Q8_0.
• Q8_0 barely fits 24GB and leaves no room for context.
• Q6_K is the sweet spot on 24GB Arc — quality near Q8, speed near Q4.

What quantization does NOT change

• Context length ceiling (determined by KV cache, separate budget).
• Tool-use reliability (mostly architecture, not precision).
• Licenses or safety properties of the base model.

Try it for yourself: Model Fit shows how the same model fits differently at each tier.