Which quantisation format to actually run.

Start here

Most of this page is arithmetic, so here is the conclusion first. Four questions settle the format, and they are asked in this order because each one can end the discussion on its own.

Everything after this explains why those four answers are what they are, and gives you the numbers to check them against your own model rather than ours. If you have not yet worked out how much memory your model needs at all, that arithmetic is a separate guide and it comes first.

Three formats, four names

There are only three number formats in common use for serving, and they differ by one thing: how many bytes each parameter costs. Everything else — AWQ, GPTQ, GGUF, bitsandbytes — is a method for producing one of these three, not a fourth format.

Read the two numeric columns as a pair. They are equal on the first two rows and they are not equal on the third, and that single asymmetry is responsible for most of the surprises further down this page.

The four names you will actually meet in a model repository map onto that table as follows.

Asking for each of them is one flag. These are the same options the vLLM guide uses at length; the point here is only how short the difference between them is.

# 1. Reference. The weights as published, nothing to prepare.
$ vllm serve meta-llama/Llama-3.3-70B-Instruct --dtype bfloat16

# 2. FP8, quantised while it loads. No second download, no calibration.
#    --kv-cache-dtype is the half everyone forgets: it is what
#    halves the CACHE as well as the weights.
$ vllm serve meta-llama/Llama-3.3-70B-Instruct \
    --quantization fp8 --kv-cache-dtype fp8

# 3. Four-bit AWQ. A DIFFERENT checkpoint, prepared by someone else.
$ vllm serve casperhansen/llama-3.3-70b-instruct-awq \
    --quantization awq_marlin

# 4. Four-bit GPTQ. Same idea, different repository and flag.
$ vllm serve TechxGenus/Llama-3.3-70B-Instruct-GPTQ \
    --quantization gptq_marlin

The half that does not shrink

Quantising weights to 4 bits does not quantise the key/value cache. AWQ and GPTQ compress weights and only weights; the cache stays at 16 bits unless you separately ask for an FP8 cache, and on a 4-bit checkpoint most people never do. That is the third column of the table above, and it is the reason the arithmetic below does not go the way anyone expects.

The consequence is easiest to see on one model. Below is Llama 3.3 70B, in 4-bit, at each context length the configurator offers — weights against cache.

The weights column never moves — that is the whole point of quantising them. The cache column grows linearly with context until, on the last row, it is larger than the model it belongs to.

Push that far enough and 4-bit stops winning at all. FP8 halves both halves; 4-bit quarters one half and leaves the other alone. So for every model there is a context length at which the two cross over, and it arrives sooner the smaller the model is — because a small model has little weight to save and the same cache growth per token.

Figures in gigabytes, for the smaller of the two formats, with the loser named underneath. Read left to right and watch the advantage drain out of the 4-bit column as the context grows.

The last row keeps its lead everywhere, because 70 billion parameters is a great deal of weight to save. The middle row ends in a dead heat: at 128k, Qwen 3 32B costs exactly the same in either format, and you would take FP8 for the quality. And the first row crosses over completely at 128k, where 4-bit asks for more memory than FP8 while also being the less faithful of the two. That combination — more memory and worse output — is the single most common quantisation mistake we see, and it is invisible if you only ever look at the size of the weights file.

What your card can actually do

FP8 is a hardware feature before it is a software one. A card whose tensor cores do not speak it will still load an FP8 checkpoint — the stack unpacks the weights to 16 bits on the way to the multiply — but you keep only the download saving and none of the speed, and an FP8 cache is not available to you at all. Four-bit is the opposite: it runs everywhere, because the weights are unpacked to 16 bits before the arithmetic regardless.

That column describes the silicon, which is not quite the same question as where FP8 is a good idea. The catalogue's own note against the format names Hopper and Blackwell, and that is a statement about the serving stacks rather than the tensor cores: those are the generations on which FP8 kernels and FP8 caches have had the most use and the fewest surprises. Ada Lovelace will run it. Hopper and Blackwell are where we would put a deployment that depends on it.

One more thing worth knowing before you pick a card. Llama 3.3 70B in FP8 at 8k needs 81.9 GB, and an 80 GB card — the NVIDIA A100 PCIe among others — is 1.9 GB short of it. Not short enough to be obvious on a spreadsheet, and quite short enough to fail on the machine. The card that clears it on its own is the next one up, and the configurator will tell you which that is before you pay rather than after.

What it buys in speed

Generating a token means reading the active weights out of memory once. Fewer bytes per weight means fewer bytes to read, which means more tokens a second — and on a single un-batched stream that relationship is very nearly linear, because there is nothing else for the card to be waiting on. This is the same memory-bandwidth ceiling the cost-per-token guide divides by, and it is deliberately conservative. Below it is applied to a model small enough to run on every single-card machine we rent, so that the format and the card can be read off the same table.

Llama 3.1 8B on one card, one stream at a time. Every machine listed holds it in all three formats, so the rows are comparable across as well as down — and the ratio between the three columns is the same on every line, because it is set by the bytes per weight and nothing else.

Two cautions on reading that table. The first is that it describes one request at a time, and almost nobody serves one request at a time: under continuous batching the weights are read once for the whole batch, so the arithmetic rather than the memory becomes the limit and the ratio between the columns narrows. The second is that dequantising costs something. Four-bit weights must be unpacked before they are multiplied, and on a small model at low batch size that unpacking can eat a visible share of what the smaller read bought you. The direction of the table is reliable; the exact multiples are not a promise.

What it costs you

This is the part of the subject where confident numbers should be distrusted, including ours. Quantisation error depends on the model far more than on the method, and the published comparisons disagree with each other because they measure different models on different tasks. What can be said honestly is the shape of it.

Which leads to the only recommendation on this page that is not arithmetic: run both. You are renting a machine with root on it and no metered anything, so serving the same model twice on two ports and pushing your own hundred prompts through each costs an evening and settles the question for your workload rather than for someone else's. That is a materially better answer than any table, this one included — and it is the reason we would rather show you the formula than a score.

Which machine that means

The reason any of this matters is the invoice. Below is the cheapest machine in our catalogue that holds each model whole at 8k context, in each of the three formats — whole meaning on one card, without splitting it, because a model split across cards brings the interconnect into the question and that is a different guide.

At 8k context, on the machines we actually rent. Where a cell names several cards, that is the smallest node the card ships in and not a split model: the biggest cards are sold in fours and eights, so the cheapest way to avoid splitting a 70B at full precision is to buy four of them and use one. That is exactly the bill quantisation exists to avoid.

Two habits make the difference between using that table well and being caught out by it. Size at the context you will actually run, not at the model's maximum — the second table on this page is what happens to people who confuse the two. And check the format against the card before you order, not after: NVIDIA RTX A6000, NVIDIA A100 PCIe and NVIDIA A100 SXM4 cannot give you the FP8 numbers no matter which flag you pass. The configurator reports the memory required and whether it lands on one card, for every node and every format, before anything is paid. If what you are weighing is renting at all against paying an API by the token, that break-even is worked out separately, and quantisation moves it a long way in the hardware's favour.