How much VRAM do I need for an LLM? Exact numbers for 8B, 14B, 32B and 70B at Q4, Q8 and FP16

If you're asking "how much VRAM do I need for a local LLM?", the answer comes from three numbers you can work out in two minutes: model weights + KV cache + runtime overhead. This guide gives the formulas, works them out for the model sizes people actually run (8B, 13B/14B, 32B, 70B), and shows how much context length adds. All numbers are worked out from the published model configs, and you can check every one yourself.

Units: GB here means 10^9 bytes. A card sold as "24 GB" really has 24 GiB ≈ 25.8 GB, which leaves a little more room than the label suggests.

The formula: how much VRAM an LLM needs

VRAM ≈ weights + KV cache + overhead

1. Weights = parameters × bytes per weight
2. FP16 / BF16: 2 bytes per weight
3. Q8_0 (8-bit GGUF): about 8.5 bits ≈ 1.06 bytes (each block of 32 weights also stores a scale)
4. Q4_K_M (the usual 4-bit GGUF): about 4.85 bits ≈ 0.6 bytes (it mixes 4- and 6-bit blocks plus scales)
5. Plain "4 bits = 0.5 bytes" is too low. Real 4-bit files are about 20% bigger.
6. KV cache = 2 × layers × KV heads × head dim × bytes per value × tokens
7. The "2" is for the keys and the values. Bytes per value is 2 at FP16 and about 1 if you turn on an 8-bit KV cache.
8. Overhead: about 1–2 GB for the CUDA/Metal context, compute buffers and, if the GPU drives your monitor, the desktop.

Sanity check against a real file: Llama 3.3 70B has 70.6B parameters. 70.6 × 0.606 = 42.8 GB, and the actual Q4_K_M GGUF is 42.5 GB. At Q8_0, 70.6 × 1.0625 = 75.0 GB, which also matches the real file.

How much VRAM for 7B and 8B models?

The KV cache for Llama 3.1 8B (32 layers, 8 KV heads, head dim 128) is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KB per token. At 8K tokens that adds 1.07 GB.

Totals at 8K context (+1.5 GB overhead): Q4 ≈ 7.4 GB, Q8 ≈ 11.1 GB, FP16 ≈ 18.6 GB. So an 8 GB card runs an 8B model at Q4 with a short-to-medium context, a 12 GB card runs it at Q8, and FP16 needs a 24 GB card.

How much VRAM for 13B and 14B models?

The two models are about the same size, but their KV caches are very different:

• Qwen2.5 14B uses grouped-query attention (48 layers, 8 KV heads): 192 KB per token, so 1.61 GB at 8K.
• Llama 2 13B has no GQA (40 layers, 40 KV heads): 800 KB per token, so 3.36 GB at its 4K maximum.

Totals: Qwen2.5 14B at 8K context comes to Q4 ≈ 12.0 GB, Q8 ≈ 18.7 GB and FP16 ≈ 32.5 GB. A 12 GB card is just enough for Q4, and a 16 GB card gives comfortable headroom. Llama 2 13B at Q4 with 4K context comes to about 12.8 GB.

How much VRAM for a 32B model at Q4 (and Q8, FP16)?

KV cache: 64 layers × 8 KV heads × 128 × 2 × 2 = 256 KB per token, so 2.15 GB at 8K.

Totals at 8K context: Q4 ≈ 23.5 GB, Q8 ≈ 38.5 GB, FP16 ≈ 69.2 GB.

Can a 24 GB GPU run a 32B model at Q4? Yes. A 24 GB card really holds about 25.8 GB, so 32B at Q4_K_M fits with roughly 8K–12K context on an FP16 KV cache, or about 16K–20K with an 8-bit KV cache. Q8 does not fit on one 24 GB card; you need about 40 GB (two 24 GB cards, or a 48 GB card).

How much VRAM for a 70B model (Llama 70B at Q4)?

KV cache: 80 layers × 8 KV heads × 128 × 2 × 2 = 320 KB per token, so 2.68 GB at 8K.

Totals at 8K context: Q4 ≈ 47 GB, Q8 ≈ 79 GB, FP16 ≈ 145 GB.

• Can a single RTX 3090/4090 (24 GB) run 70B? Not fully on the GPU. About half the Q4 model would sit in system RAM, so generation drops to a few tokens per second, because it's limited by RAM bandwidth.
• What fits 70B Q4 fully on the GPU? Plan for 48 GB for short context: two 24 GB cards, or one 48 GB workstation card. Apple Silicon with 64 GB+ unified memory also holds it.

How much VRAM does context length add? (KV cache examples)

This is the part people underestimate. The KV cache grows linearly with context, and the table uses an FP16 cache:

Qwen2.5 is natively 32K and needs RoPE scaling (YaRN) to go to 128K. The numbers show what that would cost.

Worked example: Llama 3.1 8B at Q4 with 128K context = 4.9 (weights) + 17.2 (KV) + 1.5 = 23.6 GB. The cache is three times the size of the model. With an 8-bit KV cache (for example --cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp, or OLLAMA_KV_CACHE_TYPE=q8_0 with flash attention in Ollama), the cache halves to about 8.6 GB and the total drops to about 15 GB.

The table also shows why the KV cache depends on architecture, not only on size. Llama 2 13B (no GQA) needs 800 KB per token, about six times the 128 KB of Llama 3.1 8B. Always read num_key_value_heads in the model's config.json before buying hardware for long context.

Quick answers: which GPU for which model?

• 8 GB VRAM: 7B–8B at Q4, up to about 8K–16K context.
• 12 GB: 8B at Q8, or 14B at Q4 with short context.
• 16 GB: 14B at Q4 with up to about 32K context, or 8B at Q8 with long context. 8B at FP16 (about 18 GB) does not quite fit.
• 24 GB: 32B at Q4 (8K–20K context depending on KV precision), or 14B at Q8.
• 48 GB: 70B at Q4 with 8K–16K context; 32B at Q8.
• 80 GB+: 70B at Q8. FP16 70B needs about 145 GB, which means multiple GPUs.

These are minimums for "it loads and runs." Speed is a separate question: tokens per second are roughly capped by memory bandwidth ÷ bytes read per token (about the size of the weights). That's why a 70B model split into system RAM crawls even when it technically fits.

Run the numbers for your own model

Do it once by hand with the formulas above, then use the free calculator for anything else: the Local LLM Memory & Speed Estimator takes model size, precision, context length, KV cache per 1K tokens and overhead, and also estimates the speed ceiling from your memory bandwidth. It runs entirely in your browser.

The same estimator is also on Hugging Face if you'd rather run it there: LLM Memory Estimator on Hugging Face Spaces.

Before you buy a GPU for local LLMs

VRAM is the first filter, but it isn't the whole decision. Power supply, PCIe lanes for a second card, case airflow, used-market risk on a 3090, and whether a unified-memory Mac beats a multi-GPU tower for your workload all matter. If you'd like a reusable way to make that call:

• Hardware Buying & System Design Toolkit: workload-first frameworks and checklists for sizing and buying a system, including AI/LLM boxes.
• HARDWARE THEORY: the ebook on how modern CPUs, GPUs and memory actually work, so specs like bandwidth and VRAM stop being marketing numbers.

Written by Yunus's assistant. Model configs are from the official config.json files on Hugging Face (layers, KV heads, head dim). File-size checks use the published Llama 3.3 70B GGUF sizes. Real usage varies by a few percent with runtime and quant variant.