Cost guide · 9 min read

# What a million tokens costs, both ways.

An API bills you per token and nothing while you are idle. A machine bills you per month and nothing per token. Which one is cheaper is a single division — and the answer depends far more on how busy you keep the machine than on the price of either.

The short answer

- **The break-even is a volume:** Not a price. **$692 a month** buys you as many tokens as the machine can actually produce — the question is whether you have that much work.
- **Against a frontier API:** A single un-batched stream on our cheapest node already costs **$6.42** per million. That is under the $15.00 a top closed model charges, before you batch anything.
- **Against the same open weights:** You need roughly **7.1×** your single-stream throughput to beat $0.90 per million. That is what continuous batching is for, and it is routine.
- **Where the API keeps winning:** Spiky traffic, low volume, and any model whose weights nobody will sell you.

## The short version

Almost every comparison of these two you will read comes out as a single number — "self-hosting is ten times cheaper", "the API is cheaper until you are enormous" — and both are the same mistake, which is quoting a cost per token for a machine that is not billed per token. A rented GPU has exactly one price and it does not move: whether you generate a billion tokens this month or none, the invoice is identical. So the honest question is not "what does a token cost here" but *"how many tokens do I need to generate before the fixed price beats the meter"*.

That number is a division, and it is done below on our own prices and our own throughput model — the same one the [configurator](https://gpuserver.io/configure) uses. Everything after it is about the two things that decide whether you can reach that volume: batching, and the hours you leave the machine idle.

## The arithmetic

Four lines. The third is the one worth writing on a whiteboard; the fourth is the one people skip, and skipping it is what turns a sound estimate into a wrong one.

Break-even volume, and what it demands of the machine

```
# 1. What the API charges. Linear in what you use, zero when you stop.
api_cost_per_month = output_tokens_per_month / 1e6 × api_price_per_million

# 2. What the machine charges. A constant. Tokens are free once you own the hours.
machine_cost_per_month = monthly_price

# 3. Set them equal. This is the break-even VOLUME, in output tokens per month.
break_even_tokens = monthly_price / api_price_per_million × 1e6

# 4. And the only question that matters: can the machine produce that many?
#    730 hours is the month we bill, so 2,628,000 seconds of it.
sustained_tokens_per_second = break_even_tokens / 2,628,000
```

Line 4 is a *sustained* rate, not a peak. A machine that hits 600 tokens a second for one hour a day and idles for the other twenty-three has a sustained rate of 25, and the invoice does not care which of the two you talk about.

## What one stream costs

Start with the worst case, because it is the only figure that can be stated without assumptions: one request at a time, no batching, the machine otherwise idle. Every node below runs Llama 3.3 70B at 4-bit (AWQ, GPTQ), and the throughput is our own conservative estimate — the one that is [bounded by memory bandwidth](https://gpuserver.io/guides/vram-sizing) rather than measured on a good day.

**Cost per million output tokens at a single un-batched stream, by node**

| Node | How it runs the model | Per month | One stream | Cost per million |
|---|---|---|---|---|
| [2 × NVIDIA RTX 5090](https://gpuserver.io/gpu/rtx-5090) 64 GB total · PCIe 5.0 ×16 | Split across all 2 cards 43.1 GB needed, 32 GB per card | $692 | 41 tok/s | $6.42 per 1M output |
| [2 × NVIDIA RTX 4090](https://gpuserver.io/gpu/rtx-4090) 48 GB total · PCIe 5.0 ×16 | Split across all 2 cards 43.1 GB needed, 24 GB per card | $416 | 23 tok/s | $6.88 per 1M output |
| [4 × NVIDIA RTX 5090](https://gpuserver.io/gpu/rtx-5090) 128 GB total · PCIe 5.0 ×16 | Split across all 4 cards 43.1 GB needed, 32 GB per card | $1,345 | 68 tok/s | $7.53 per 1M output |
| [4 × NVIDIA RTX 4090](https://gpuserver.io/gpu/rtx-4090) 96 GB total · PCIe 5.0 ×16 | Split across all 4 cards 43.1 GB needed, 24 GB per card | $814 | 38 tok/s | $8.15 per 1M output |
| [2 × NVIDIA A100 PCIe](https://gpuserver.io/gpu/a100-40gb) 80 GB total · PCIe 5.0 ×16 | Split across all 2 cards 43.1 GB needed, 40 GB per card | $802 | 36 tok/s | $8.48 per 1M output |
| [8 × NVIDIA RTX 5090](https://gpuserver.io/gpu/rtx-5090) 256 GB total · PCIe 5.0 ×16 | Split across all 8 cards 43.1 GB needed, 32 GB per card | $2,584 | 100 tok/s | $9.83 per 1M output |

Read this as a ceiling, never as a quote. It is what a token costs if you make the machine answer one question at a time and leave it alone the rest of the month — the least efficient way to own a GPU that exists. Every real serving stack does better, and the next two sections are about how much better.

## The break-even, per tier

Take the cheapest row above — [2 × NVIDIA RTX 5090](https://gpuserver.io/gpu/rtx-5090) at $692 a month — and ask how much work it has to do to beat each tier of the API market.

**Break-even monthly volume and the sustained throughput it requires**

| What you would pay instead | Per 1M output | Break-even volume | Sustained rate | Versus one stream |
|---|---|---|---|---|
| Frontier proprietary model The best closed model of the moment. Nobody rents you its weights. | $15.00 | 46Mtokens a month | 18 tok/saround the clock | Already cheaper |
| Mid-tier proprietary model The workhorse tier most products actually ship on. | $4.00 | 173Mtokens a month | 66 tok/saround the clock | 1.6× of single-stream |
| Open-weight 70B, specialist provider Same weights The same weights you would run yourself, served by someone else. | $0.90 | 769Mtokens a month | 293 tok/saround the clock | 7.1× of single-stream |
| Open-weight 70B, cheapest serverless Same weights The floor of the market, usually with a shared queue behind it. | $0.40 | 1,730Mtokens a month | 658 tok/saround the clock | 16.1× of single-stream |

The last column is the whole article. Where it says "already cheaper", one un-batched stream on a $692 machine beats the API and there is nothing to think about. Where it shows a multiple, the machine can still win — but only if you actually feed it, which is the next section.

**The two proprietary rows are not a fair fight, and they are the fight most people are actually in.** You cannot rent the weights of a frontier model, so "self-host it instead" is not an option — the real comparison is *a closed model at $15.00 against an open one at $6.42 that you own outright*, and whether the open one is good enough for your task. That is a quality question, not a cost question, and it is the one worth spending an afternoon on before any of this arithmetic matters.

## Batching is the whole game

Generating one token requires reading the active weights out of memory once. Generating a hundred tokens for a hundred *different* requests requires reading them once as well — the same weights serve every request in the batch. That is why a serving stack with continuous batching produces many times the tokens per second of a single stream on identical hardware, and why the ratio in the last column of that table is reachable at all.

### What batching gives you

Aggregate throughput climbs steeply with concurrency while the weights stay the bottleneck, then flattens as the KV cache fills the card and the arithmetic itself becomes the limit.

- Large multiples over one stream are ordinary, not optimistic
- It costs you nothing except the memory the extra caches take
- [vLLM does it by default](https://gpuserver.io/guides/serve-llama-70b) — you do not configure it, you feed it

### What it costs you

Throughput does not scale linearly with concurrency, and per-request latency gets worse as the batch grows. Sixty-four streams do not give you sixty-four times the tokens.

- Each stream sees slower tokens than it would alone
- The KV cache is what runs out first — [size it deliberately](https://gpuserver.io/guides/vram-sizing)
- An idle batch slot generates nothing: concurrency you do not have is worth nothing

The practical consequence: treat the multiple in the break-even table as a *throughput target*, not a user count. A target of ten times single-stream does not mean "ten users"; it means the server has to average ten times its un-batched rate, which usually takes rather more than ten concurrent requests and rather fewer than a hundred. Measure it on the machine before you commit to a number — it is one benchmark, and it settles the argument.

## The idle hours you still pay for

Everything above assumed the load is spread evenly across the month. It never is. A machine rented by the month is paid for on Sunday at 04:00, and the tokens you did not generate then do not carry forward. Divide the week you actually serve into the week you actually pay for:

**How a duty cycle multiplies the throughput you need while busy**

| When the load arrives | Hours a week | Peak rate needed | Effective cost per million |
|---|---|---|---|
| Continuously, 24/7 An endpoint that answers whenever it is called | 168 h | The sustained rate | $6.42 at one stream |
| Twelve hours a day A product with one hemisphere of users | 84 h | 2.0× while it is busy | $12.84 at one stream |
| Working hours, weekdays An internal tool used by your own team | 40 h | 4.2× while it is busy | $26.97 at one stream |
| Two hours a day A batch job, or a demo that gets opened sometimes | 14 h | 12.0× while it is busy | $77.07 at one stream |

The third row is where most internal tools live, and it quietly multiplies your break-even by more than four. That is the single most common reason a self-hosting estimate turns out wrong — not the token price, not the hardware, but the assumption that a working week is a whole week.

## Why your prompt is nearly free

One asymmetry works strongly in favour of owning the machine, and no price-per-token comparison shows it. An API charges you for input tokens — usually a quarter to a third of the output price, per token. On your own GPU, input tokens are processed in the prefill pass, which handles the entire prompt in parallel and is limited by arithmetic rather than by memory bandwidth. Prefill runs at an order of magnitude more tokens per second than generation does.

**The more context you send, the better self-hosting looks.** A retrieval-augmented request with a 20,000-token prompt and a 300-token answer is dominated by input on an API invoice and by output on your own machine. Two workloads with identical monthly bills at one provider can be two-to-one apart at the other, in the direction of the hardware. If your prompts are long — RAG, document extraction, long-context summarisation — run the arithmetic on *your* input-to-output ratio rather than on output alone.

## What the arithmetic leaves out

Four things that never appear in a cost-per-token table, and that decide the question at least as often as the numbers do.

**Every request goes somewhere.** An API sees your prompts, and your prompts are your product, your customers' documents, or your code. On a machine you rent, the weights and the traffic stay on the box — and [we never asked who you are](https://gpuserver.io/no-kyc-gpu-server) in the first place. That is not a line item, but for some workloads it is the entire decision.

**Models get deprecated; yours does not.** A hosted model can change, get a new version with different behaviour, or be retired on a schedule that is not yours. A set of weights on your own NVMe behaves the same in a year, which matters a great deal if you have evaluated against it.

**A fixed price is a different kind of number.** Metered billing means a bug, a retry loop, or a burst of traffic turns into an invoice you find out about afterwards. A monthly term cannot surprise you: the worst case is that the machine is slow, not that it is expensive.

**Somebody has to run it.** The honest cost of self-hosting includes an afternoon of setup and the occasional evening when a driver or a container misbehaves. Our [documentation](https://gpuserver.io/docs) exists to make that an afternoon rather than a week, but it is not zero, and pretending otherwise is how these comparisons lose their credibility.

## Which side you are on

In order, and stop at the first one that describes you.

1. **You need a frontier closed model's quality.** Then this article does not apply: nobody rents you those weights. Use the API, and revisit when an open model closes the gap on your specific task.
2. **Your volume is low or you do not know it yet.** Use the API. It is the cheapest way to find out what your traffic actually looks like, and the meter is a perfectly good measuring instrument.
3. **Your traffic is spiky and tolerates a queue.** Use the API, or split it: your own machine for the steady base load, a hosted endpoint for the peaks. Nothing forces the decision to be all one way.
4. **You are sustaining real, batched volume against an open-weight model.** This is the case the hardware wins, and it wins by a lot — the fixed price stops moving while the meter keeps running. Check your own number in the table above before you believe it.
5. **The prompts are sensitive, the model must not change, or the bill must be knowable in advance.** Then the arithmetic is a formality. Rent the machine that holds your model, and pay for it [without telling anyone who you are](https://gpuserver.io/guides/pay-in-crypto).

Whichever line you stopped on, the number to check is the one in the break-even table for *your* model and *your* volume. The [configurator](https://gpuserver.io/configure) reports, for every node we rent, whether your model fits on a single card, has to be split, and what the throughput estimate is — which is every input this arithmetic needs. If you are still deciding between renting by the month and metering by the hour, that is [a different break-even](https://gpuserver.io/guides/monthly-vs-hourly), and it is worked out too.

## Work out your own break-even on a real machine.

The configurator gives you the throughput estimate and the monthly price for every node in the catalogue — the two numbers this guide divides.

[Open the configurator](https://gpuserver.io/configure) [Read the guides](https://gpuserver.io/guides)

## Other guides

- [Practice · 11 min Serving Llama 3.3 70B on one node From a delivered machine to an OpenAI-compatible endpoint, with the flags that matter and the two that silently halve your throughput. Read the guide](https://gpuserver.io/guides/serve-llama-70b)
- [Practice · 10 min Fine-tuning a 70B model on one card QLoRA on a single 48 GB GPU: what fits, what it costs for a month, and why full fine-tuning is a different order of machine. Read the guide](https://gpuserver.io/guides/lora-finetune)
- [Payment · 8 min Paying for a server in crypto What actually happens between clicking pay and getting root, which coin to choose, and the four mistakes that lose money on a first payment. Read the guide](https://gpuserver.io/guides/pay-in-crypto)

---

Source: https://gpuserver.io/guides/api-vs-self-hosting/. This file is generated from the same data as the website; if a figure here differs from a page, the page is authoritative and this file is stale — the canonical source is https://gpuserver.io/.