How many people one GPU can actually serve.

The short version

Every guide that tells you whether a model fits answers a question about one request. A product is not one request. The moment two people use your endpoint at the same time, the machine needs a second copy of the conversation in memory — and a third, and a fortieth — while the weights themselves are stored exactly once.

So the number that decides how many people you can serve is not the size of the card. It is the size of the card minus the model, divided by what one conversation costs. Both of those terms are known before you order anything, which makes capacity one of the few things in machine learning you can work out on paper and be right about.

Everything below runs the same arithmetic as the configurator, on the same catalogue, at 4-bit with 8k of context. If the memory formula itself is new to you, it has a guide of its own — this page is what happens to it when more than one person shows up.

What is left after the weights

Take the reference model of this site and put it on every size of node built from the same card. Same silicon, same price per card, same everything — only the amount of memory changes. Watch the last two columns move at completely different speeds.

The first row is the whole lesson: a card that cannot even hold the weights serves nobody, and it is not close. The rows after it gain memory in equal steps and gain seats in accelerating ones, because the 40 GB toll is paid on the first row and never again.

What one user actually costs

Now the same models, sorted the way a buyer sorts them: cheapest first. The last column is the monthly price divided by the number of people the machine can hold at once — the only column that compares two machines honestly when you are building something more than a demo.

One assumption, stated because it changes every number: each seat count is for one instance of the model spread across the whole node, which is what --tensor-parallel-size gives you. The weights are stored once and every card contributes its remaining memory to the same pool of conversations. Run two separate copies on the same node instead and you pay the toll twice, for fewer seats in total.

Read the two coloured cells against their price column. NVIDIA L40S costs $714 a month and holds 2; 4 × NVIDIA L4 costs $564 — less money — and holds 19. That is 10× the seats for 0.79× the invoice, and the expensive one is the one most shortlists keep, because it is the one where the model fits on a single card.

Across the whole catalogue rather than the first ten rows, the best price per seat for this model is 8 × NVIDIA RTX A6000 at $2,239 a month: 119 concurrent requests, $18.82 each. It is a far larger invoice than anything on the shortlist above, and it is still the cheapest way to seat a user — which is the sentence that decides whether you are sizing a product or a prototype.

Context length is the other multiplier

Everything above assumed 8k of context. That assumption is doing more work than the choice of machine. The cache is paid per token as well as per user, so the same node seats a completely different number of people depending on how long you let a conversation get.

One machine, four models, four context lengths. The node is 8 × NVIDIA L4 at $1,106 a month — the cheapest in our catalogue that holds all four models at once, so that every number below is measured against the same memory.

On one unchanged machine, Llama 3.3 70B goes from 105 concurrent users to 3 — a factor of 35× — purely because of a number you set on the command line. Nothing about the hardware moved.

Two things follow. The first is that the context column you should read is the one you actually serve, not the one the model card advertises — a model that supports 128k does not oblige you to reserve it. The second is that the cache per token varies enormously between models of similar size, because it is set by the attention design rather than the parameter count; the largest models on that table have some of the smallest caches, which is the single most counter-intuitive fact in this whole area.

Four ways to buy back capacity

In descending order of how much they give you for how little they cost. The first one is nearly free and nearly always left on the table.

Same node, same weights, same money. The only change is the precision the cache is stored in, and it is the difference between a machine that serves a team and a machine that serves a product.

Seats are not the same as service

One correction before you size anything on the numbers above, and it is the correction that keeps this page honest. Everything here counts how many conversations the machine can hold open. It does not promise that all of them are being answered quickly.

Memory and bandwidth are two different ceilings. On 8 × NVIDIA L4, Llama 3.3 70B has room for 52 concurrent conversations, while a single user alone on that machine sees roughly 17 tokens a second. Filling all 52 seats does not give each of them 17 tokens a second — generation shares the same memory bandwidth, so the aggregate rises with batching and the per-user rate falls. The bandwidth ceiling is reached well before the memory one.

The good news is that the two ceilings move together: the machines with a large amount of memory left over after the weights are, with few exceptions, the machines with more bandwidth as well. Choosing on headroom rarely costs you speed. Setting the serving stack up on the machine you end up with is a separate guide, and the flags it covers are where these numbers are cashed in.

Choosing for your own user count

In order. Stop at the first line that describes your load.

1. One request at a time. Batch jobs, an internal tool, a single developer. Buy the cheapest machine that holds the model on one card and stop reading — every column on this page is answering a question you do not have.
2. A handful of people, occasionally. A team tool, an early product. Work out your peak concurrent requests, not your user count: a hundred registered users rarely means more than a handful of simultaneous ones. Then take the cheapest machine whose seat count is comfortably above that peak.
3. A real product with real traffic. Sort the catalogue by price per seat rather than by price, cap your context to what you serve, turn the cache down to 8 bits, and expect the winner to be a machine you would not have shortlisted on its monthly price.
4. Long documents or long conversations. Read the context table first and the machine table second. At 32k and above the cache dominates so completely that the model choice matters less than the attention design behind it.
5. Spiky, unpredictable traffic. Size for the peak you must not fail, because the cache cannot be borrowed when it runs out — a request that has nowhere to put its conversation waits in a queue. If the peak is rare and enormous, a per-token API for the overflow is cheaper than a machine sized for a spike that happens twice a month.

Whichever line you stopped on, do the subtraction before you do anything else: take the memory of the machine, remove the weights of your model in the precision you will serve, and look at what is left. That remainder is the product you are renting. The configurator runs the same arithmetic across every node we operate, and what a month of it costs is worked out separately.