Will It Fit? The Complete VRAM Calculation Formula for Local LLMs

The most fundamental question before downloading any model or buying computer hardware is: will it fit in VRAM?

Running out of VRAM (Out of Memory, or OOM) causes crashes, system freezes, or forces the inference engine to fall back to sluggish CPU execution. To determine whether a model will run natively on your graphics card, you cannot simply look at the file size on disk.

Total memory footprint consists of three distinct mathematical terms:

$$\text{Total VRAM Required} = \text{Model Weights} + \text{KV Cache Memory} + \text{Runtime Overhead}$$

This guide breaks down each term in the equation so you can calculate memory requirements with complete precision.

Term 1: Model weights memory ($M_{\text{weights}}$)

The static memory required to store the neural network’s parameters. This depends on the total parameter count and the quantization format:

$$M_{\text{weights}} = \text{Parameters (billions)} \times \frac{\text{Bits Per Weight (bpw)}}{8} \times 1.05$$

(The 1.05 multiplier accounts for unquantized metadata, normalization layer weights, and quantization scaling constants).

Common bits-per-weight reference values:

• FP16 / BF16 (Unquantized): 16.0 bpw (2.0 bytes/param)
• Q8_0 (8-bit Quant): 8.5 bpw (~1.06 bytes/param)
• Q6_K (6-bit Quant): 6.56 bpw (~0.82 bytes/param)
• Q5_K_M (5-bit Quant): 5.5 bpw (~0.69 bytes/param)
• Q4_K_M (4-bit Quant): 4.5 bpw (~0.56 bytes/param)
• Q3_K_M (3-bit Quant): 3.5 bpw (~0.44 bytes/param)

Example calculation:

For a 30-billion parameter model at Q4_K_M: $$M_{\text{weights}} = 30 \times \frac{4.5}{8} \times 1.05 \approx 17.7\text{ GB}$$

Term 2: The Key-Value (KV) cache memory ($M_{\text{kv}}$)

As an LLM generates tokens or digests a prompt, it must store past attention keys and values for every layer of the network. Unlike static model weights, KV cache scales linearly with active context length.

For standard Grouped-Query Attention (GQA) architectures (used by Llama 3, Qwen 2.5/3, Mistral), the formula is:

$$M_{\text{kv}} = 2 \times n_{\text{layers}} \times n_{\text{kv_heads}} \times d_{\text{head}} \times L_{\text{context}} \times B_{\text{precision}}$$

Where:

• $2$ accounts for both Keys and Values.
• $n_{\text{layers}}$ is the number of transformer layers in the model.
• $n_{\text{kv_heads}}$ is the number of key-value attention heads (typically 4 to 8 in modern GQA models).
• $d_{\text{head}}$ is the dimension per attention head (typically 128).
• $L_{\text{context}}$ is the active context length in tokens (e.g., 4,096, 8,192, or 32,768).
• $B_{\text{precision}}$ is the storage precision per element (2 bytes for FP16, 1 byte for 8-bit quantized cache, 0.5 bytes for 4-bit cache).

Context scaling reference table (Standard 16-bit FP16 KV Cache):

Notice: Pushing a 70B model to 32K context requires over 10GB of VRAM strictly for the conversation context, completely separate from the weights!

Term 3: Runtime and execution overhead ($M_{\text{runtime}}$)

The inference engine (llama.cpp, vLLM, SGLang, or OpenVINO) must allocate memory buffers for execution:

• CUDA/Vulkan Context: 300MB – 600MB reserved by the graphics driver upon initialization.
• Compute Scratchpads & Activation Tensors: 400MB – 800MB required to store intermediate layer activations during the forward pass.
• Serving Buffer Margin: Engines with continuous batching (like vLLM) reserve additional memory for token block allocation tables.

Rule of thumb: Always add 1.0 GB to 1.5 GB of overhead for standalone inference, and 2.0 GB for multi-user web serving.

Putting it together: The complete calculation

Let us calculate the total footprint for running Qwen3-Coder-30B-A3B at Q4_K_M with 8,192 context tokens:

1. Weights: $30 \times (4.5 / 8) \times 1.05 = 17.7\text{ GB}$
2. KV Cache (8K): $2 \times 48 \times 8 \times 128 \times 8,192 \times 2\text{ bytes} \approx 1.6\text{ GB}$
3. Runtime Overhead: $\approx 1.0\text{ GB}$

$$\mathbf{Total\ Memory} = 17.7 + 1.6 + 1.0 = \mathbf{20.3\text{ GB}}$$

The verdict on hardware fit:

• On a 16GB GPU (e.g. RTX 4060 Ti 16GB): Will not fit at 8K context. You must either reduce context length to 2K tokens or offload ~5GB of layers to system RAM.
• On a 24GB GPU (e.g. Arc B60 Pro or RTX 3090): Fits completely in VRAM with ~3.7GB of safety headroom! In our lab benchmark bm-001, this exact setup consumed 18.1GB of allocated VRAM and sustained 38.6 tokens/sec.

Quick VRAM fit lookup by GPU capacity

• 8GB VRAM: Fits 3B to 8B models at Q4_K_M with 4K context.
• 12GB VRAM: Fits 8B to 14B models natively at Q4/Q6 with 8K context. Runs 30B models via partial CPU layer offload.
• 16GB VRAM: Fits up to 14B models at Q8_0, or 27B models at tight Q3_K_M.
• 24GB VRAM: The gold standard for consumer AI. Fits 30B to 35B models natively at Q4_K_M with 8K context, or 70B models with heavy offload.
• 32GB+ (Dual GPU / Mac Unified Memory): Fits 70B models at Q4_K_M and enables deep context windows (32K+).

For a complete model-by-model reference table covering dozens of architectures, visit the canonical VRAM Lookup Hub. To test your exact hardware configuration interactively, use the Model Fit Estimator.