Sizing guide · 10 min read

# How many people one GPU can actually serve.

A machine that holds your model is not a machine that serves your users. The weights are a toll, paid once; what is left after them is the entire budget you have for serving people. That leftover — not the size of the card — decides your capacity, and it moves far faster than the memory does.

The short answer

- **Capacity is the leftover, not the card:** **Llama 3.3 70B** at 4-bit reserves **40 GB** before it serves anyone. Every concurrent request after that costs **2.9 GB** more.
- **Which is why memory pays twice:** On one and the same card, going from 48 GB to 240 GB multiplies the memory by 5.0× and the number of seats by **35×**. The weights are already paid for.
- **The worst buy on the shortlist:** [NVIDIA L40S](https://gpuserver.io/gpu/l40s) at **$714** a month holds 2 concurrent requests — **$357.00** per seat.
- **The best one costs less:** [4 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) at **$564** holds 19 — **$29.68** per seat, on a cheaper invoice.

## The short version

Every guide that tells you whether a model *fits* answers a question about one request. A product is not one request. The moment two people use your endpoint at the same time, the machine needs a second copy of the conversation in memory — and a third, and a fortieth — while the weights themselves are stored exactly once.

So the number that decides how many people you can serve is not the size of the card. It is the size of the card *minus* the model, divided by what one conversation costs. Both of those terms are known before you order anything, which makes capacity one of the few things in machine learning you can work out on paper and be right about.

**The weights are a toll.** Paid once, on entry, whatever your traffic. They do not grow with your users and they never come back.

**The key/value cache is the rent.** Paid per concurrent request, for as long as that request is open, and proportional to how long the conversation is.

**Capacity is what is left, divided by the rent.** Which is why a machine with twice the memory very often serves ten times the people rather than twice as many.

**And why the cheapest machine that fits is usually the worst value.** It fits because it has almost nothing left over, and what it has left over is the only part you are actually buying.

Everything below runs the same arithmetic as the [configurator](https://gpuserver.io/configure), on the same catalogue, at 4-bit with 8k of context. If the memory formula itself is new to you, [it has a guide of its own](https://gpuserver.io/guides/vram-sizing) — this page is what happens to it when more than one person shows up.

## What is left after the weights

Take the reference model of this site and put it on every size of node built from the same card. Same silicon, same price per card, same everything — only the amount of memory changes. Watch the last two columns move at completely different speeds.

**Concurrent requests for Llama 3.3 70B at 4-bit and 8k of context, on every node built from the same card**

| Node | Memory | Left after the weights | Concurrent requests | Per month |
|---|---|---|---|---|
| [NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) | 24 GB | — | does not hold it | $125 |
| [2 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) | 48 GB | 8 GB | 2 | $285 |
| [4 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) | 96 GB | 56 GB | 19 | $564 |
| [8 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) | 192 GB | 152 GB | 52 | $1,106 |
| [10 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) | 240 GB | 200 GB | 69 | $1,371 |

The first row is the whole lesson: a card that cannot even hold the weights serves nobody, and it is not close. The rows after it gain memory in equal steps and gain seats in accelerating ones, because the 40 GB toll is paid on the first row and never again.

**The arithmetic, in one line.** `seats = (memory − weights) ÷ cache_per_request`. For Llama 3.3 70B at 4-bit that is 40 GB of toll and 2.9 GB of rent per open conversation. Subtract before you divide — doing it the other way round is how a machine gets bought for a demo and discovered in production.

## What one user actually costs

Now the same models, sorted the way a buyer sorts them: cheapest first. The last column is the monthly price divided by the number of people the machine can hold at once — the only column that compares two machines honestly when you are building something more than a demo.

One assumption, stated because it changes every number: each seat count is for *one* instance of the model spread across the whole node, which is what `--tensor-parallel-size` gives you. The weights are stored once and every card contributes its remaining memory to the same pool of conversations. Run two separate copies on the same node instead and you pay the toll twice, for fewer seats in total.

**The ten cheapest nodes that hold Llama 3.3 70B, with what each one costs per concurrent user**

| Node | Memory | Per month | Concurrent requests | Per user |
|---|---|---|---|---|
| [2 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) 2 cards · 24 GB each | 48 GB | $285 | 2 | $142.50 |
| [NVIDIA RTX A6000](https://gpuserver.io/gpu/rtx-a6000) One card · 48 GB | 48 GB | $286 | 2 | $143.00 |
| [2 × NVIDIA RTX 4090](https://gpuserver.io/gpu/rtx-4090) 2 cards · 24 GB each | 48 GB | $416 | 2 | $208.00 |
| [4 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) 4 cards · 24 GB each | 96 GB | $564 | 19 | $29.68 |
| [2 × NVIDIA RTX A6000](https://gpuserver.io/gpu/rtx-a6000) 2 cards · 48 GB each | 96 GB | $597 | 19 | $31.42 |
| [2 × NVIDIA RTX 5090](https://gpuserver.io/gpu/rtx-5090) 2 cards · 32 GB each | 64 GB | $692 | 8 | $86.50 |
| [NVIDIA L40S](https://gpuserver.io/gpu/l40s) One card · 48 GB | 48 GB | $714 | 2 | $357.00 |
| [2 × NVIDIA A100 PCIe](https://gpuserver.io/gpu/a100-40gb) 2 cards · 40 GB each | 80 GB | $802 | 13 | $61.69 |
| [4 × NVIDIA RTX 4090](https://gpuserver.io/gpu/rtx-4090) 4 cards · 24 GB each | 96 GB | $814 | 19 | $42.84 |
| [NVIDIA A100 PCIe](https://gpuserver.io/gpu/a100-80gb) One card · 80 GB | 80 GB | $1,091 | 13 | $83.92 |

Read the two coloured cells against their price column. NVIDIA L40S costs $714 a month and holds 2; 4 × NVIDIA L4 costs $564 — *less money* — and holds 19. That is 10× the seats for 0.79× the invoice, and the expensive one is the one most shortlists keep, because it is the one where the model fits on a single card.

**This is not an argument against single-card machines.** A model that fits on one card needs no sharding, no interconnect, no tensor-parallel flags, and it answers a lone user faster than the same model spread over four cards. If your load is one request at a time — a batch job, an internal tool, a demo — that machine is the right buy and the per-user column is irrelevant to you. The column only starts mattering when people arrive at the same time, and it matters enormously from that point on.

Across the whole catalogue rather than the first ten rows, the best price per seat for this model is [8 × NVIDIA RTX A6000](https://gpuserver.io/gpu/rtx-a6000) at $2,239 a month: 119 concurrent requests, $18.82 each. It is a far larger invoice than anything on the shortlist above, and it is still the cheapest way to seat a user — which is the sentence that decides whether you are sizing a product or a prototype.

## Context length is the other multiplier

Everything above assumed 8k of context. That assumption is doing more work than the choice of machine. The cache is paid per token as well as per user, so the same node seats a completely different number of people depending on how long you let a conversation get.

One machine, four models, four context lengths. The node is [8 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) at $1,106 a month — the cheapest in our catalogue that holds all four models at once, so that every number below is measured against the same memory.

**Concurrent requests on one node, by model and context length, at 4-bit**

| Model | Cache per token | 4k | 8k | 32k | 128k |
|---|---|---|---|---|---|
| Llama 3.1 8B 4 GB of weights | 128 KiB | 325 | 162 | 40 | 10 |
| Qwen 3 32B 16 GB of weights | 256 KiB | 150 | 75 | 18 | 4 |
| Llama 3.3 70B 35 GB of weights | 320 KiB | 105 | 52 | 13 | 3 |
| Qwen 3 235B-A22B (MoE) 118 GB of weights | 188 KiB | 67 | 33 | 8 | 2 |

On one unchanged machine, Llama 3.3 70B goes from 105 concurrent users to 3 — a factor of 35× — purely because of a number you set on the command line. Nothing about the hardware moved.

Two things follow. The first is that the context column you should read is the one you actually serve, not the one the model card advertises — a model that *supports* 128k does not oblige you to reserve it. The second is that the cache per token varies enormously between models of similar size, because it is set by the attention design rather than the parameter count; [the largest models on that table have some of the smallest caches](https://gpuserver.io/guides/mixture-of-experts), which is the single most counter-intuitive fact in this whole area.

## Four ways to buy back capacity

In descending order of how much they give you for how little they cost. The first one is nearly free and nearly always left on the table.

Cap the declared context Free, and the largest win Serving stacks reserve cache for the maximum length you declare, not the length you use. Declaring 128k and serving 8k throws away sixteen times the memory you needed — and that memory was your entire seating capacity. Set `--max-model-len` to your real ceiling on day one.

Quantise the cache to 8 bits Roughly doubles the seats Quantising the *weights* to 4 bits does nothing to the cache: it stays in 16 bits in every common stack unless you ask. Asking is one flag, the quality cost is usually invisible on chat workloads, and the table below is what it returns.

Quantise the weights further Helps once, then stops Halving the weights hands the difference straight to the cache, so it buys seats on a crowded machine. But it is a one-off gain against a fixed toll, and it costs quality — [which format costs how much](https://gpuserver.io/guides/awq-vs-gptq-vs-fp8) is a decision of its own.

Rent the leftover, not the card The structural fix Every lever above works on the margin. Moving to a node whose memory exceeds your weights by a wide margin changes the shape of the problem, and it is the only one of the four that keeps paying as you grow.

**Concurrent requests on the same node with a 16-bit cache and with an 8-bit cache, at 8k of context**

| Model | 16-bit cache | 8-bit cache | Seats gained |
|---|---|---|---|
| Llama 3.1 8B | 162 | 325 | +163 |
| Qwen 3 32B | 75 | 150 | +75 |
| Llama 3.3 70B | 52 | 105 | +53 |
| Qwen 3 235B-A22B (MoE) | 33 | 67 | +34 |

Same node, same weights, same money. The only change is the precision the cache is stored in, and it is the difference between a machine that serves a team and a machine that serves a product.

## Seats are not the same as service

One correction before you size anything on the numbers above, and it is the correction that keeps this page honest. Everything here counts how many conversations the machine can hold *open*. It does not promise that all of them are being answered quickly.

Memory and bandwidth are two different ceilings. On [8 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4), Llama 3.3 70B has room for 52 concurrent conversations, while a single user alone on that machine sees roughly 17 tokens a second. Filling all 52 seats does not give each of them 17 tokens a second — generation shares the same memory bandwidth, so the aggregate rises with batching and the per-user rate falls. The bandwidth ceiling is reached well before the memory one.

**Use the seat count as a ceiling, not as a target.** A machine loaded to its last seat is a machine where everyone is waiting. The seat count tells you which hardware is even capable of your concurrency — it eliminates the machines that cannot, which is most of the shortlist — and then you measure the rate you actually want per user and back off from the ceiling. Sizing to the ceiling is the second most common way to buy the wrong machine, right after ignoring the cache entirely.

The good news is that the two ceilings move together: the machines with a large amount of memory left over after the weights are, with few exceptions, the machines with more bandwidth as well. Choosing on headroom rarely costs you speed. [Setting the serving stack up](https://gpuserver.io/guides/serve-llama-70b) on the machine you end up with is a separate guide, and the flags it covers are where these numbers are cashed in.

## Choosing for your own user count

In order. Stop at the first line that describes your load.

1. **One request at a time.** Batch jobs, an internal tool, a single developer. Buy the cheapest machine that holds the model on one card and stop reading — every column on this page is answering a question you do not have.
2. **A handful of people, occasionally.** A team tool, an early product. Work out your peak concurrent requests, not your user count: a hundred registered users rarely means more than a handful of simultaneous ones. Then take the cheapest machine whose seat count is comfortably above that peak.
3. **A real product with real traffic.** Sort the catalogue by price per seat rather than by price, cap your context to what you serve, turn the cache down to 8 bits, and expect the winner to be a machine you would not have shortlisted on its monthly price.
4. **Long documents or long conversations.** Read the context table first and the machine table second. At 32k and above the cache dominates so completely that the model choice matters less than the attention design behind it.
5. **Spiky, unpredictable traffic.** Size for the peak you must not fail, because the cache cannot be borrowed when it runs out — a request that has nowhere to put its conversation waits in a queue. If the peak is rare and enormous, [a per-token API for the overflow](https://gpuserver.io/guides/api-vs-self-hosting) is cheaper than a machine sized for a spike that happens twice a month.

Whichever line you stopped on, do the subtraction before you do anything else: take the memory of the machine, remove the weights of your model in the precision you will serve, and look at what is left. That remainder is the product you are renting. The [configurator](https://gpuserver.io/configure) runs the same arithmetic across every node we operate, and [what a month of it costs](https://gpuserver.io/guides/monthly-vs-hourly) is worked out separately.

## Size the machine on the users, not on the model.

The configurator holds every node we rent, runs the same memory arithmetic as this page, and shows what is left after your model is loaded — before you pay for anything.

[Open the configurator](https://gpuserver.io/configure) [Read the guides](https://gpuserver.io/guides)

## Other guides

- [Cost · 6 min When monthly rental beats per-hour The break-even worked out on real numbers, the three costs an hourly price hides until the invoice, and the idle-time trap that triples an estimate. Read the guide](https://gpuserver.io/guides/monthly-vs-hourly)
- [Cost · 9 min Self-hosting against a per-token API The break-even between paying an API per token and renting a GPU, worked out on our own prices and throughput — and the four things the arithmetic leaves out. Read the guide](https://gpuserver.io/guides/api-vs-self-hosting)
- [Practice · 11 min Serving Llama 3.3 70B on one node From a delivered machine to an OpenAI-compatible endpoint, with the flags that matter and the two that silently halve your throughput. Read the guide](https://gpuserver.io/guides/serve-llama-70b)

---

Source: https://gpuserver.io/guides/concurrent-users/. This file is generated from the same data as the website; if a figure here differs from a page, the page is authoritative and this file is stale — the canonical source is https://gpuserver.io/.