How much VRAM a model actually needs.

The formula

There is no rule of thumb here that survives contact with a second model. The whole calculation is three lines, and it is worth doing them rather than trusting a multiplier.

# 1. Weights
weights_gb   = parameters_in_billions × bytes_per_parameter

# 2. KV cache, per token — then multiplied by the context you serve
bytes_per_tok = 2 × layers × kv_heads × head_dim × cache_bytes
cache_gb      = bytes_per_tok × context_tokens × batch / 1024³

# 3. Everything else
total_gb      = (weights_gb + cache_gb) × 1.15

The 2 is because there are two tensors per token, K and V. bytes_per_parameter is 2 for BF16, 1 for FP8, 0.5 for 4-bit. cache_bytes is a separate number, and that separation is the single most common source of a wrong answer — see below.

Weights

The easy half. It is exactly linear in the parameter count, and the only decision is precision.

Four-bit quantisation costs quality, and how much depends on the model far more than on the method. It is the right trade when it is the difference between one card and four; it is a poor trade when you are already comfortable.

The KV cache

The hard half, and the one that decides whether a machine is adequate at 4k and hopeless at 128k. It depends on layers and KV heads — not on how large the model is.

Read the last column before the first. At 4k context the cache is a rounding error on every model here; at 128k it is larger than the weights of a 70B model in 4-bit. DeepSeek V3 is the outlier because its latent attention compresses the cache by design — which is why it is usable at long context despite being the largest model on the list.

Where estimates go wrong

Worked examples

Total memory required, weights plus cache plus overhead, at 8k context and one request. This is the same calculation the configurator runs against every configuration in the catalogue.

The last column is the cheapest node in our catalogue that holds the model at 4-bit and 8k context, with the single-stream generation rate that memory bandwidth allows. Throughput figures are deliberately conservative and calibrated against published measurements — treat them as a floor, not a promise.

Serving more than one request

This is where a machine sized for a demo becomes a machine that cannot serve a product. Weights are paid once; cache is paid per concurrent sequence.

Llama 3.3 70B at 4-bit is 35 GB of weights whatever you do. Everything above 35 GB in that table is cache. This is why "it ran fine on my laptop" and "it fell over in production" are the same model on the same card.

Which machine that means

Three practical rules that follow from everything above, in the order they matter.

1. One card beats several, whenever it is possible. No sharding, no synchronisation between layers, no interconnect to think about. If your model at your context fits on a single card, buy that card.
2. Size for your real context and your real concurrency, not for the model's advertised maximum. The advertised maximum is a capability, not a requirement.
3. When you must shard, prefer NVLink. Splitting across four PCIe cards works; splitting across four NVLink cards works and is meaningfully faster. The next guide is about exactly when that difference is worth paying for.

The configurator runs this calculation live against all 41 configurations: pick a model, a precision and a context length, and it tells you which nodes hold it, which hold it on a single card, and what each one costs per month. It uses the formula on this page, so if you disagree with our arithmetic you can now say exactly where.