Buying a graphics card for gaming is about rasterization frame rates and ray tracing cores. Buying a GPU for local AI is an entirely different engineering calculation.
When running local LLMs, 90% of your user experience is dictated by two physical properties:
- VRAM Capacity (Gigabytes): Dictates whether a model fits in memory. If a model does not fit, it fails or slows down to a crawl.
- Memory Bandwidth (Gigabytes per Second): Dictates your token generation speed. In autoregressive inference, the GPU must sweep every parameter weight into compute cores for every token produced.
This guide categorizes the consumer GPU market into actionable purchasing tiers based on empirical benchmark testing from our lab.
The Four Buying Tiers
Tier 1: Budget Entry Tier ($250 – $350)
Target Workload: 8B to 14B models natively; 30B models with layer offloading.
For years, entering local AI required at least $600. The arrival of 12GB budget cards established a reliable $300 floor:
- Intel Arc B580 12GB (~$300 new): The modern budget king. Offers 12GB of VRAM and robust Vulkan performance for approximately $300. In
llama.cpp, it runs 8B models (like Llama 3.1) at 50+ tokens/sec, and handles 30B models (like Qwen3-Coder-30B) via CPU layer offload. - NVIDIA RTX 3060 12GB (~$260 – $300 used): The dependable CUDA workhorse. While its raw compute and memory bandwidth (360 GB/s) are modest, its 12GB frame buffer and flawless compatibility with all CUDA-only research libraries make it the safest plug-and-play budget card for Linux and Windows.
Avoid in this tier: Any 8GB graphics card (e.g. RTX 4060 8GB, RX 7600 8GB). 8GB is dead silicon for modern local AI; it severely restricts context windows and restricts you to tiny 3B–7B models.
Tier 2: The 24GB Workstation Sweet Spot ($650 – $800)
Target Workload: 30B to 35B models natively at full context; daily-driver coding assistants; interactive agent loops.
If you do daily software development or run autonomous agent loops, 24GB of VRAM is the optimal price-to-performance threshold:
- Used NVIDIA RTX 3090 24GB (~$700 – $750 used): The undisputed value champion of local AI. Despite being launched in 2020, its massive 24GB GDDR6X frame buffer and blistering 936 GB/s memory bandwidth allow it to generate 30B models at over 50 tokens/sec. It supports FlashAttention-2, TensorRT-LLM, and vLLM without configuration friction.
- Intel Arc B60 Pro 24GB (~$650 new): The workstation dark horse. Provides 24GB of VRAM in a single slot-friendly power profile. In our lab benchmarks (bm-001), it generated Qwen3-Coder-30B at 38.6 tokens/sec over Vulkan, and jumped to ~68 tok/s when running Intel’s OpenVINO Model Server (bm-012).
- AMD Radeon RX 7900 XT 20GB (~$680 new): Excellent high-bandwidth alternative (800 GB/s). In our benchmarks (bm-007), it recorded 269.01 tokens/sec on 8B models, though its 20GB buffer leaves slightly less margin than 24GB cards for deep context 30B models.
Tier 3: High-End Flagship ($900 – $1,800)
Target Workload: 70B models (via dual-GPU or high quantization); instantaneous code generation; multi-tenant homelab servers.
- NVIDIA RTX 4090 24GB (~$1,600 – $1,800): Maximum single-card performance. With 1,008 GB/s memory bandwidth and fourth-generation Tensor Cores, it delivers unmatched token generation rates. However, at more than double the price of a used RTX 3090, you are paying a massive premium for speed rather than added memory capacity.
- AMD Radeon RX 7900 XTX 24GB (~$900 – $950 new): High-speed 24GB workstation card (960 GB/s bandwidth). On Linux with ROCm, it approaches RTX 4090 inference speeds on supported models for nearly half the price.
- Dual Used RTX 3090 24GB (~$1,400 – $1,500 total): The ultimate enthusiast setup. Two used 3090s combined via PCIe tensor parallelism yield 48GB of total VRAM, enabling full native execution of 70B-class models (like Llama 3.3 70B) at ~20 tokens/sec.
Tier 4: The Mac / Unified Memory Alternative ($1,200 – $3,900)
Target Workload: Massive 70B+ models and 100k+ context windows at ultra-low power.
If your goal is running large 70B models or massive document ingestion pipelines without building a power-hungry desktop tower, Apple Silicon remains a compelling alternative:
- Mac Studio (M2 / M4 Max with 64GB or 128GB Unified Memory): Because system RAM is unified and directly accessible by the Metal GPU cores, a 128GB Mac can load an entire 70B model with zero offload penalties while drawing under 120W of wall power.
- The Tradeoff: While memory capacity is vast, memory bandwidth on base models (300–400 GB/s) is lower than discrete high-end GPUs, resulting in generation speeds of 15–25 tokens/sec rather than 50+ tok/s.
The Five Golden Rules of Buying a Local AI GPU
- VRAM capacity is non-negotiable: Core count and clock speeds only determine how fast a token is generated; VRAM determines if the model can run at all. Never sacrifice memory capacity for higher core clocks.
- Memory bandwidth is king for generation: Once a model fits in VRAM, tokens-per-second is directly proportional to memory bandwidth. A card with 900 GB/s bandwidth will generate roughly 2× faster than a card with 450 GB/s bandwidth.
- Check physical clearance and power requirements: High-end 24GB cards (like the RTX 3090) draw 350W+ and occupy three full PCIe slots. Ensure your power supply is at least 750W–850W and your chassis has adequate airflow.
- Beware of PCIe lane bottlenecks in multi-GPU builds: If you plan to run two cards, ensure your motherboard and CPU support at least PCIe 4.0 x8/x8 bifurcation. Dropping to PCIe x4 cuts tensor parallel communication throughput during multi-card inference.
- Never buy an 8GB card for modern LLMs: 8GB cards are obsolete for productive 2026 workflows. The minimum practical floor is 12GB; the ideal sweet spot is 24GB.
For deep architectural benchmarks across these cards, read Best GPUs for local AI, measured and Best GPUs for local AI in late 2026. To calculate the exact memory required for your preferred models, use our Model Fit tool.