GPUs for local AI, compared

At 4-bit, a model needs about 0.6 GB per billion parameters, plus the context cache and about 1 GB for the runtime. A 8 B model fits in 8 GB, a 27–32 B model needs about 20–24 GB, a 70 B model needs about 45 GB (two 24 GB cards). The “Largest model” column does this sum for each card at 8K context.

When a model writes, every new word reads all the active weights from memory once. So speed is roughly bandwidth divided by the size of the active weights. That is why an RTX 3090 (936 GB/s) still writes faster than many newer mid-range cards.

For local LLMs it is often the best value: 24 GB and high bandwidth. It uses a lot of power (350 W) and has no warranty when bought used. Check the price against a new 16 GB card: 16 GB is not enough for 27–32 B models at 4-bit with a useful context.

NVIDIA has the widest software support (CUDA). AMD cards work well with llama.cpp, Ollama and LM Studio (ROCm or Vulkan) and often give more memory for the money. Apple Silicon and AMD Ryzen AI Max machines share memory between CPU and GPU, so they can load very large models, but they write more slowly than a big desktop GPU.

Two cards add up their memory, so a 70 B model fits on two 24 GB cards. Speed does not double: the cards take turns, so you get about the speed of one card, a little less. Choose “2 cards” above to see the numbers.

They are estimates: memory bandwidth × 78% ÷ data read per word, plus a fixed cost per word (about 1 ms, 2.8 ms for mixture-of-experts models, more on Apple), at 4-bit (Q4_K_M) with an 8K context. We checked the formula against 44 real llama.cpp tests: the median error is 9%, and 42 of 44 are within ±30%. Real numbers also depend on the software version, drivers (on AMD, Vulkan is often faster than ROCm) and settings. Use them to compare cards, not as exact figures.