Quantization is the single most important technique in local AI. It is the reason a 30-billion parameter model—which originally requires 60GB of memory in 16-bit floating point precision—can comfortably execute on a consumer graphics card with 24GB or even 12GB of VRAM.
However, quantization is not magic. It represents a precise mathematical compromise between memory footprint, inference generation speed, and output quality. This guide breaks down the underlying mechanics, explains the modern GGUF quantization formats, and examines what our benchmark lab measured when sweeping precision levels on real hardware.
How quantization works: Compressing weights
When a neural network is trained, its weights and activation tensors are stored as high-precision 16-bit floating point numbers (FP16 or BF16), where each individual parameter consumes 2 bytes of storage:
$$\text{FP16 Memory} = \text{Parameters} \times 2\text{ bytes}$$
For a 30B model, storing the raw weights alone requires: $$30 \times 10^9 \times 2\text{ bytes} = 60\text{ GB of VRAM}$$
Because consumer GPUs top out at 12GB to 24GB of VRAM, running full FP16 models locally is physically impossible without tens of thousands of dollars in enterprise server hardware.
Quantization solves this by mapping high-precision continuous floating-point values into compact, discrete integer representations (typically 4-bit, 5-bit, 6-bit, or 8-bit integers). A 4-bit integer uses only 0.5 bytes per parameter—compressing the memory footprint by roughly 70% to 75%.
$$\text{Quantized Memory} = \text{Parameters} \times \frac{\text{Bits per Weight}}{8} + \text{Quantization Scales Overhead}$$
The modern GGUF k-quant hierarchy
Historically, simple integer quantization rounded all weights uniformly, causing significant degradation in sensitive layers. The modern llama.cpp standard uses k-quants (K-means quantization), which selectively allocate higher bit precision to critical transformer blocks (such as attention matrices) while aggressively compressing less sensitive feed-forward layers.
| Quant Type | Average Bits/Weight | Relative Size (vs FP16) | Perplexity Impact | Practical Role |
|---|---|---|---|---|
| Q8_0 | 8.5 bpw | 50% | Negligible (<0.1%) | The “near-lossless” reference benchmark |
| Q6_K | 6.56 bpw | 38% | Imperceptible (<0.5%) | Quality enthusiast sweet spot; plenty of headroom |
| Q5_K_M | 5.5 bpw | 32% | Minimal (<1.0%) | Excellent balance of intelligence and memory fit |
| Q4_K_M | 4.5 bpw | 26% | Small (1–3%) | The universal daily driver default |
| Q3_K_M | 3.5 bpw | 21% | Moderate (5–10%) | Emergency fit for fitting larger models into tight VRAM |
| Q2_K | 2.5 bpw | 15% | Severe (>20%) | High hallucination rate; generally avoid |
The _M suffix denotes “medium” (quantizing key attention tensors at higher bits while quantizing value tensors lower). _S denotes “small” (more aggressive compression across all tensors).
What our lab measured: The empirical quantization sweep
Rather than relying on theoretical perplexity scores, our benchmark lab ran an identical model—Qwen3-Coder-30B-A3B—across Q4, Q6, and Q8 precision levels on our testbed system (“Jitori”, equipped with an Intel Arc B60 Pro 24GB running Vulkan).
In benchmark bm-003, we measured decode speed, actual VRAM consumption, and functional coding capability on the HumanEval benchmark:
| Precision Level | Decode Speed (tok/s) | VRAM Footprint | HumanEval pass@1 Score | Tradeoff Summary |
|---|---|---|---|---|
| Q4_K_M | 38.6 tok/s | 18.1 GB | 0.81 | Fast interactive speed; fits within 24GB comfortably |
| Q6_K | 27.4 tok/s | 22.6 GB | ≈0.84 | 29% slower decode; uses 4.5GB more memory for ~3pts coding lift |
| Q8_0 | 19.8 tok/s | 26.0 GB | 0.86 | 49% slower decode; overflows 24GB cards into system RAM |
Source: Measured in benchmark bm-003. HumanEval scores represent directional pass@1 code completion.
Core findings from the data
- Throughput scales inversely with bit precision: Memory bandwidth is the primary bottleneck during LLM generation. Because the GPU must stream every weight matrix from VRAM into the compute cores for each generated token, reducing weight size from Q8 (26GB) to Q4 (18GB) almost doubles generation speed (from 19.8 tok/s to 38.6 tok/s).
- The quality penalty is surprisingly small: On standard software engineering evaluations, dropping from Q8_0 down to Q4_K_M resulted in a modest 5-point delta (0.86 down to 0.81). The model remains fully capable of generating correct syntax, structuring functions, and resolving algorithmic challenges.
- Q4_K_M is the rational default: Running Q8_0 pushes memory usage to 26GB, forcing partial layer offloading or requiring multi-GPU configurations, all while cutting tokens-per-second in half. For daily interactive coding and chat, Q4_K_M offers the optimal balance of speed and capability.
Practical rules for selecting your quant
When downloading GGUF models from Hugging Face or Ollama, follow this simple hierarchy:
- If the model fits your VRAM at Q4_K_M with 2–4GB to spare: Run Q4_K_M. It maximizes generation speed and allows room for deep context windows.
- If you have ample excess VRAM headroom: Step up to Q5_K_M or Q6_K. You gain a modest improvement in nuanced reasoning with minimal speed degradation.
- Never drop to Q3 or Q2 to run a model that is too large for your hardware: An 8B model running cleanly at Q4_K_M will almost always outperform a heavily compressed 14B model squashed down to Q2_K, while hallucinating significantly less.
For exact memory requirements across different model families and context windows, consult the VRAM Lookup table and test your configuration in Model Fit.