Will it fit? VRAM math for local LLMs

The arithmetic behind Model Fit, with worked examples. Once you can do this in your head, you stop needing the tool.

The formula

Memory footprint, simplified:

footprint = weights + kv_cache + runtime_overhead

weights_gb       = params_in_billions × bytes_per_param(quant)
kv_cache_gb      = (context_tokens / 1000) × 0.18   # conservative
runtime_overhead = ~1.0 GB                            # llama.cpp baseline

Worked example: Qwen3-Coder-30B-A3B at Q4_K_M, 8K context

weights       = 30 × 0.55         = 16.5 GB
kv_cache      = 8.192 × 0.18      = 1.47 GB
runtime       =                   1.0 GB
-----------------------------------------
total                            ≈ 19.0 GB

On a 24GB card (Arc B60 Pro, RX 7900 XTX, RTX 4090), this fits with ~5GB headroom — "comfortably" in our estimator. Our measured result was 18.1 GB VRAM used, close to the estimate.

Worked example: 70B at Q4_K_M, 8K context

weights       = 70 × 0.55         = 38.5 GB
kv_cache      = 8.192 × 0.18      = 1.47 GB
runtime       =                   1.0 GB
-----------------------------------------
total                            ≈ 41.0 GB

Doesn't fit any single consumer 24GB card. Options: (a) multi-GPU split across two 24GB cards, (b) drop to Q3_K_M (~31.5GB — still over 24GB, requires offload), (c) accept RAM offload with major speed penalty.

FAQ

How accurate is the VRAM footprint formula?

It is a rule of thumb, accurate to within about 1-2GB for dense models. Our measured Qwen3-Coder-30B-A3B at Q4_K_M came in at 18.1GB used versus an estimate of about 19.0GB. Real measurements always override the estimate.

Can I run a 70B model on a single 24GB card?

No. A 70B model at Q4_K_M with 8K context needs about 41GB. Your options are multi-GPU split across two 24GB cards, dropping to Q3_K_M (still over 24GB, requires offload), or accepting RAM offload with a major speed penalty.

What does the formula ignore?

Layer count, attention implementation (MHA vs GQA vs MLA), batch size, OS VRAM usage, and model-specific architecture like MoE expert routing. Use Model Fit for a quick estimate, then confirm against a real benchmark.

Ready to plug in your numbers? Open Model Fit →