Sizing guide · 9 min read

# How much VRAM a model actually needs.

Two numbers decide it: the weights, which are fixed, and the KV cache, which grows with how much context you serve. Almost every wrong answer comes from estimating the second one from the first.

The short answer

- **Weights:** Parameters in billions × bytes per parameter. 70B at 4-bit is **35 GB**.
- **KV cache:** 2 × layers × KV heads × head dim × bytes, per token. Nothing to do with parameter count.
- **Overhead:** Add about **15%** for activations, the CUDA context and allocator fragmentation.
- **The trap:** Quantising weights to 4-bit does **not** quantise the cache. It stays FP16 in every common stack.

## The formula

There is no rule of thumb here that survives contact with a second model. The whole calculation is three lines, and it is worth doing them rather than trusting a multiplier.

Total VRAM, in GB

```
# 1. Weights
weights_gb   = parameters_in_billions × bytes_per_parameter

# 2. KV cache, per token — then multiplied by the context you serve
bytes_per_tok = 2 × layers × kv_heads × head_dim × cache_bytes
cache_gb      = bytes_per_tok × context_tokens × batch / 1024³

# 3. Everything else
total_gb      = (weights_gb + cache_gb) × 1.15
```

The `2` is because there are two tensors per token, K and V. `bytes_per_parameter` is 2 for BF16, 1 for FP8, 0.5 for 4-bit. `cache_bytes` is a *separate* number, and that separation is the single most common source of a wrong answer — see [below](https://gpuserver.io/guides/vram-sizing#wrong).

## Weights

The easy half. It is exactly linear in the parameter count, and the only decision is precision.

**Weight memory by precision, for a selection of model sizes**

| Model | BF16 / FP16 | FP8 | 4-bit (AWQ, GPTQ) |
|---|---|---|---|
| Llama 3.1 8B 8B parameters | 16.0 GB | 8.0 GB | 4.0 GB |
| Qwen 3 32B 32B parameters | 64.0 GB | 32.0 GB | 16.0 GB |
| Llama 3.3 70B 70B parameters | 140.0 GB | 70.0 GB | 35.0 GB |
| Llama 3.1 405B 405B parameters | 810.0 GB | 405.0 GB | 202.5 GB |

Four-bit quantisation costs quality, and how much depends on the model far more than on the method. It is the right trade when it is the difference between one card and four; it is a poor trade when you are already comfortable.

## The KV cache

The hard half, and the one that decides whether a machine is adequate at 4k and hopeless at 128k. It depends on layers and KV heads — *not* on how large the model is.

**Two models of very different sizes, side by side.** Llama 3.3 70B has 80 layers and 8 KV heads, so 320 KB per token. Gemma 3 27B — a third of the parameters — has 62 layers and 16 KV heads, so 496 KB per token, which is 1.6× *more*. Any estimate scaled from parameter count gets this backwards.

**KV cache size by model and context length, at FP16 cache precision, one request**

| Model | Per token | 4k context | 8k context | 32k context | 128k context |
|---|---|---|---|---|---|
| Llama 3.1 8B 32 layers · 8 KV heads | 128 KB | 0.5 GB | 1.0 GB | 4.0 GB | 16.0 GB |
| Gemma 3 27B 62 layers · 16 KV heads | 496 KB | 1.9 GB | 3.9 GB | 15.5 GB | 62.0 GB |
| Llama 3.3 70B 80 layers · 8 KV heads | 320 KB | 1.3 GB | 2.5 GB | 10.0 GB | 40.0 GB |
| DeepSeek V3 671B-A37B (MoE) Latent attention | 70 KB | 0.3 GB | 0.5 GB | 2.2 GB | 8.8 GB |
| Llama 3.1 405B 126 layers · 8 KV heads | 504 KB | 2.0 GB | 3.9 GB | 15.8 GB | 63.0 GB |

Read the last column before the first. At 4k context the cache is a rounding error on every model here; at 128k it is larger than the weights of a 70B model in 4-bit. DeepSeek V3 is the outlier because its latent attention compresses the cache by design — which is why it is usable at long context despite being the largest model on the list.

## Where estimates go wrong

**Scaling the cache from parameter count.** The most common error, and the one shown above. A 27B model can need more cache than a 70B one.

**Assuming 4-bit weights mean a 4-bit cache.** They do not. AWQ and GPTQ quantise weights only; the cache stays FP16 unless you explicitly enable FP8 KV. A 70B model at 4-bit and 128k context is 35 GB of weights and 40 GB of cache.

**Sizing for one request.** The cache is per concurrent sequence. Serving eight users at once needs eight times the cache — see [below](https://gpuserver.io/guides/vram-sizing#batch).

**Forgetting the overhead.** Activations, the CUDA context and allocator fragmentation are real. 15% is a deliberately cautious allowance; ignoring it entirely is how a model that "fits" fails to load.

**Counting total VRAM across cards.** Four 24 GB cards are not one 96 GB card. Tensor parallelism can split a model across them, but every layer then synchronises over the link — see the [NVLink guide](https://gpuserver.io/guides/nvlink-vs-pcie).

**Sizing a MoE by its active parameters.** DeepSeek V3 activates 37B per token but you must hold all 671B in memory. Active parameters predict *speed*; total parameters decide whether it loads at all.

## Worked examples

Total memory required, weights plus cache plus overhead, at 8k context and one request. This is the same calculation the configurator runs against every configuration in the catalogue.

**Total VRAM required at 8k context, by model and precision**

| Model | BF16 / FP16 | FP8 | 4-bit (AWQ, GPTQ) | Smallest node that fits |
|---|---|---|---|---|
| Llama 3.1 8B 8B parameters | 20 GB | 10 GB | 6 GB | [NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $125/mo · on one card · ~34 tok/s |
| Mistral Small 24B 24B parameters | 57 GB | 28 GB | 15 GB | [NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $125/mo · on one card · ~11 tok/s |
| Gemma 3 27B 27B parameters | 67 GB | 33 GB | 20 GB | [NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $125/mo · on one card · ~10 tok/s |
| Qwen 3 32B 32B parameters | 76 GB | 38 GB | 21 GB | [NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $125/mo · on one card · ~8 tok/s |
| Llama 3.3 70B 70B parameters | 164 GB | 82 GB | 43 GB | [2 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $285/mo · split across the node · ~7 tok/s |
| Mixtral 8×22B (MoE) 39B active of 141B | 326 GB | 163 GB | 83 GB | [4 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $564/mo · split across the node · ~4 tok/s |
| Qwen 3 235B-A22B (MoE) 22B active of 235B | 542 GB | 271 GB | 137 GB | [8 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $1,106/mo · split across the node · ~10 tok/s |
| DeepSeek V3 671B-A37B (MoE) 37B active of 671B | 1,544 GB | 772 GB | 386 GB | [8 × NVIDIA A100 PCIe](https://gpuserver.io/gpu/a100-80gb) $7,906/mo · split across the node · ~37 tok/s |
| Llama 3.1 405B 405B parameters | 936 GB | 468 GB | 237 GB | [10 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $1,371/mo · split across the node · ~3 tok/s |

The last column is the cheapest node in our catalogue that holds the model at 4-bit and 8k context, with the single-stream generation rate that memory bandwidth allows. Throughput figures are deliberately conservative and calibrated against published measurements — treat them as a floor, not a promise.

## Serving more than one request

This is where a machine sized for a demo becomes a machine that cannot serve a product. Weights are paid once; cache is paid per concurrent sequence.

**Total VRAM for Llama 3.3 70B at 4-bit, by concurrent requests and context**

| Concurrent requests | 4k context | 8k context | 32k context |
|---|---|---|---|
| 1 request | 42 GB one 80 GB card | 43 GB one 80 GB card | 52 GB one 80 GB card |
| 4 requests | 46 GB one 80 GB card | 52 GB one 80 GB card | 86 GB one H200 |
| 16 requests | 63 GB one 80 GB card | 86 GB one H200 | 224 GB multi-GPU node |
| 64 requests | 132 GB one H200 | 224 GB multi-GPU node | 776 GB multi-GPU node |

Llama 3.3 70B at 4-bit is 35 GB of weights whatever you do. Everything above 35 GB in that table is cache. This is why "it ran fine on my laptop" and "it fell over in production" are the same model on the same card.

**Two ways to buy back cache memory.** Enable FP8 KV cache if your stack supports it — that halves the numbers above at a quality cost that is usually invisible. And cap `--max-model-len` at the context you actually serve: vLLM reserves cache for the declared maximum, so declaring 128k when you serve 8k throws away sixteen times the memory you need.

## Which machine that means

Three practical rules that follow from everything above, in the order they matter.

1. **One card beats several, whenever it is possible.** No sharding, no synchronisation between layers, no interconnect to think about. If your model at your context fits on a single card, buy that card.
2. **Size for your real context and your real concurrency**, not for the model's advertised maximum. The advertised maximum is a capability, not a requirement.
3. **When you must shard, prefer NVLink.** Splitting across four PCIe cards works; splitting across four NVLink cards works and is meaningfully faster. The [next guide](https://gpuserver.io/guides/nvlink-vs-pcie) is about exactly when that difference is worth paying for.

The [configurator](https://gpuserver.io/configure) runs this calculation live against all 41 configurations: pick a model, a precision and a context length, and it tells you which nodes hold it, which hold it on a single card, and what each one costs per month. It uses the formula on this page, so if you disagree with our arithmetic you can now say exactly where.

## Check your own model against every machine we rent.

The configurator does the arithmetic above for all 41 configurations, before you pay anything.

[Open the configurator](https://gpuserver.io/configure) [Read the guides](https://gpuserver.io/guides)

## Other guides

- [Sizing · 7 min NVLink or PCIe: which one your job needs When the link between cards decides your throughput, when it changes nothing, and how to tell which case you are in before you pay for the wrong node. Read the guide](https://gpuserver.io/guides/nvlink-vs-pcie)
- [Sizing · 9 min Choosing a card for image and video models What FLUX, SDXL, SD 3.5 and Wan 2.1 need in VRAM, which card in our catalogue holds each, and why the cheapest one that fits is rarely the one to rent. Read the guide](https://gpuserver.io/guides/gpu-for-flux-sdxl)
- [Sizing · 10 min Choosing a quantisation format What AWQ, GPTQ, GGUF and FP8 each cost in memory, speed and quality — and why the format with the smallest weights rarely gives the smallest model. Read the guide](https://gpuserver.io/guides/awq-vs-gptq-vs-fp8)

---

Source: https://gpuserver.io/guides/vram-sizing/. This file is generated from the same data as the website; if a figure here differs from a page, the page is authoritative and this file is stale — the canonical source is https://gpuserver.io/.