Sizing guide · 10 min read

# What a mixture of experts really costs.

A name like “235B-A22B” contains two numbers, and almost everyone reads the wrong one. One of them decides what you rent; the other decides how fast it answers. They can be a factor of ten apart.

The short answer

- **Total parameters decide the machine:** **DeepSeek V3 671B-A37B (MoE)** at 4-bit needs **386 GB** with 8k of context. A dense model the size of its *active* parameters would need 22 GB — **17.6×** less.
- **Active parameters decide the speed:** Only a fraction of the weights is read per token, so it generates faster than a dense model of the same mass — but the routing takes a large part of that advantage straight back.
- **The cheapest node that holds it:** [8 × NVIDIA A100 PCIe](https://gpuserver.io/gpu/a100-80gb) at **$7,906** a month. It is almost never the one to rent.
- **The one worth renting:** [4 × NVIDIA B200 SXM6](https://gpuserver.io/gpu/b200) at **$11,790** — 1.5× the money for 2.8× the throughput.

## The short version

A mixture of experts splits the feed-forward part of each layer into many small networks and sends every token through only a few of them. The published name records both facts: the total number of parameters, and the number that are active for any one token. *You must hold all of the first in memory, and you only read the second per token.* That single sentence is the whole guide, and getting it backwards is the most expensive mistake we see people make before an order.

The consequence is a model with a strange cost shape: it rents like a giant and runs like a mid-size model. Whether that is a bargain or a trap depends entirely on which of the two numbers your bottleneck is.

**Sizing the machine?** Use the total. Every expert has to be resident, because you cannot know in advance which ones the next token will need. Routing is decided per token, at run time.

**Estimating the speed?** Start from the active count, then take a large discount. Generation reads only the active weights, which is the whole point — but the router, the expert dispatch and the traffic between cards are not free, and they scale badly across a node.

**Serving long context or many users?** Look at the attention design, not at the parameter counts. The largest models on this page have the *smallest* key/value cache per token, and past a certain length that reverses the ranking entirely.

**Just want the quality?** A dense model that fits on one card is easier, cheaper and faster to operate than a mixture of experts split across eight. Reach for a MoE when nothing dense gives you the answer you need, not because the name looks impressive.

The arithmetic below is the same one the [configurator](https://gpuserver.io/configure) runs, on the same catalogue. If you have not yet worked out how much memory a model needs at all, [that formula comes first](https://gpuserver.io/guides/vram-sizing) — this page is what changes once the model is a mixture of experts.

## Reading the name

Three naming conventions are in circulation and they all encode the same pair of numbers, which is why the confusion survives. Once you can read them, the sizing question answers itself.

235B-A22B Total, then active The clearest of the three. 235 billion parameters live in memory; 22 billion of them are read for any given token. The letter A means *active*, and it is the number that does **not** tell you what to rent.

671B-A37B Same convention, larger At this size the gap becomes absurd on purpose: the active share is under six per cent. Nothing about the machine you need follows from the 37.

8×22B Experts, then expert size The older convention, and the one that misleads most. It does not mean 176 billion parameters, because attention and embeddings are shared rather than duplicated; and it does not mean 22 billion either, because two experts are active per token, not one.

A22B, A37B An average, not a guarantee The active count is what routing costs on typical text. It is not a cap: a batch whose tokens happen to spread across many experts touches more of the model, which is one reason measured throughput sits below the arithmetic.

**The mixture-of-experts models in our sizing catalogue, total against active parameters**

| Model | Total parameters | Active per token | Active share |
|---|---|---|---|
| Mixtral 8×22B (MoE) 56 layers | 141 B | 39 B | 28 % |
| Qwen 3 235B-A22B (MoE) 94 layers | 235 B | 22 B | 9 % |
| DeepSeek V3 671B-A37B (MoE) 61 layers | 671 B | 37 B | 6 % |
| Llama 3.3 70B Dense, for comparison | 70 B | 70 B | 100 % |

The last column is the one to remember. A dense model reads everything it stores; a mixture of experts reads a slice. Everything strange about these models — good and bad — comes out of that one line.

## The memory you actually rent

Here is the whole trap in one table. The middle column is what the model needs on the machine; the column after it is what a reader who took the active number at face value would have budgeted for. The last column is the bill for that misreading.

**VRAM needed at 4-bit with 8k of context, against what the active-parameter count suggests**

| Model | Weights at 4-bit | Total needed | If it were its active size | Overshoot |
|---|---|---|---|---|
| Mixtral 8×22B (MoE) | 71 GB | 83 GB at 8k | 24 GB | 3.4× |
| Qwen 3 235B-A22B (MoE) | 118 GB | 137 GB at 8k | 14 GB | 9.5× |
| DeepSeek V3 671B-A37B (MoE) | 336 GB | 386 GB at 8k | 22 GB | 17.6× |

Read the last column as a shopping error. Someone sizing on the active count alone would go looking for a single card and end up needing a whole node — not a slightly bigger machine, a different category of machine, at a different price.

**No, you cannot keep the unused experts on disk.** It is the first idea everybody has, and serving stacks do offer it. The problem is that routing is decided per token: the set of experts you need changes several times per generated word, so a card that holds a subset spends its time faulting weights across PCIe instead of computing. The result is not a slightly slower model, it is one that runs at a small fraction of its speed. Offloading is a way to make a model *run* on a machine that cannot hold it, not a way to *serve* it.

## What the active parameters buy

Generation is limited by memory bandwidth: each new token requires re-reading the weights that participate in it. A mixture of experts reads only its active slice, so the ceiling on its speed is set by a much smaller number than its size suggests. That part of the promise is real, and it is why these models exist at all.

The table below puts every model on the *same* machine — [8 × NVIDIA A100 PCIe](https://gpuserver.io/gpu/a100-80gb) at $7,906 a month, the cheapest node in our catalogue that holds all of them at once. Comparing throughput figures taken on different machines compares nothing.

**Single-stream generation on one node, mixtures of experts against dense models**

| Model | Total | Read per token | Estimated tokens/s |
|---|---|---|---|
| DeepSeek V3 671B-A37B (MoE) Mixture of experts · split across 8 cards | 671 B | 18.5 GB | 37 one stream |
| Llama 3.1 405B Dense · split across 8 cards | 405 B | 202.5 GB | 19 one stream |
| Qwen 3 235B-A22B (MoE) Mixture of experts · split across 8 cards | 235 B | 11.0 GB | 62 one stream |
| Mixtral 8×22B (MoE) Mixture of experts · split across 8 cards | 141 B | 19.5 GB | 35 one stream |
| Llama 3.3 70B Dense · on one card | 70 B | 35.0 GB | 25 one stream |

The two rows worth comparing are the largest mixture of experts and the largest dense model, because both are split across the whole node. The mixture is the bigger model by a wide margin and it still generates faster, since it reads a fraction of itself per token — that is the trade the architecture exists to make. Read the rows marked *on one card* differently: those models occupy a single card of the node and leave the other seven free, which is a cheaper machine waiting to be chosen, not a slower one.

**These figures already carry a large penalty, and they should.** Pure active-parameter arithmetic badly over-predicts a mixture of experts. Our first version of this estimator did exactly that, and it was wrong by roughly a factor of five against published measurements for the largest model on this page. Routing costs a pass of its own, experts have to exchange activations between cards at every layer, and the batch spreads across more experts than the average implies. The estimator now applies a deliberately conservative correction, calibrated on measured figures rather than chosen — which is why it reads low rather than high. Treat every number here as a floor to beat on the machine, not a target to plan against.

The second thing the arithmetic hides is that a mixture of experts is harder to shard well. A dense model split across cards synchronises once per layer; a mixture also has to route tokens to whichever card holds the chosen expert, which is a different and less predictable traffic pattern. This is one of the few workloads where [the interconnect genuinely earns its price](https://gpuserver.io/guides/nvlink-vs-pcie) rather than merely appearing on the invoice.

## The half that gets cheaper

Everything so far has been bad news for the big models. Here is the compensation, and it is a large one that almost no comparison mentions: the models with the most parameters on this page have the *least* key/value cache per token. The cache is set by the attention design — layers, key/value heads, head dimension — and it has nothing to do with how many experts sit behind attention.

**Key/value cache per token, and what one stream costs at each context length**

| Model | Cache per token | 4k | 8k | 32k | 128k |
|---|---|---|---|---|---|
| Mixtral 8×22B (MoE) 8 KV heads | 224 KiB | 0.9 GB | 1.8 GB | 7.0 GB | 28.0 GB |
| Qwen 3 235B-A22B (MoE) 4 KV heads | 188 KiB | 0.7 GB | 1.5 GB | 5.9 GB | 23.5 GB |
| DeepSeek V3 671B-A37B (MoE) Latent attention | 70 KiB | 0.3 GB | 0.5 GB | 2.2 GB | 8.8 GB |
| Llama 3.3 70B 8 KV heads | 320 KiB | 1.3 GB | 2.5 GB | 10.0 GB | 40.0 GB |
| Llama 3.1 405B 8 KV heads | 504 KiB | 2.0 GB | 3.9 GB | 15.8 GB | 63.0 GB |

Those columns are per *concurrent request*. Multiply by the number of people using the endpoint at once and the ranking in the last column decides your machine far more than the weights do.

**DeepSeek V3 671B-A37B (MoE) carries the smallest cache on the list, at 70 KiB a token.** Against Llama 3.3 70B at 320 KiB, that is a factor of **4.6×** in the direction of the larger model. It compresses the cache by projecting keys and values into a small shared latent vector instead of storing them per head. So the model that costs the most to load is the one that costs the least to keep busy — which is why it stays usable at long context, and why the arithmetic flips once you serve more than a handful of people at once.

The practical reading: weights are a fixed admission fee, cache is the running cost. A mixture of experts charges an enormous admission fee and then a small running cost per user. A dense model of comparable ability is the other way round. Which of the two is cheaper for you is decided by concurrency and context length, not by the parameter count on the model card.

## Which machines hold them

At 4-bit, with the reference context, and taking the cheapest node in our catalogue that holds each model — whether on one card or split across the node. These are the entry tickets, not recommendations; the next section explains why.

**The cheapest configuration we rent that holds each mixture-of-experts model**

| Model | Needs | Cheapest node that holds it | Per month | Tokens/s |
|---|---|---|---|---|
| Mixtral 8×22B (MoE) | 83 GB | [4 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) 96 GB total · 24 GB per card | $564 | 4 |
| Qwen 3 235B-A22B (MoE) | 137 GB | [8 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) 192 GB total · 24 GB per card | $1,106 | 10 |
| DeepSeek V3 671B-A37B (MoE) | 386 GB | [8 × NVIDIA A100 PCIe](https://gpuserver.io/gpu/a100-80gb) 640 GB total · 80 GB per card | $7,906 | 37 |

Every one of these is a multi-card node, and that is the honest headline of this page: nothing in our catalogue holds one of these models on a single card, however small the active count makes them look. [The full catalogue](https://gpuserver.io/#catalog) has the rest of the configurations, and [the configurator](https://gpuserver.io/configure) will run this same check against your own model and context.

## The cheapest node is the wrong one

Fitting is a threshold, not a goal. Once a model is split across a node, throughput depends on the memory bandwidth of the cards it is split across — and bandwidth per dollar varies by more than a factor of two across configurations that all clear the same memory bar. Here is DeepSeek V3 671B-A37B (MoE) on every node that holds it, ordered by price, with the only column that matters on the right.

**Every node that holds DeepSeek V3 671B-A37B (MoE), with the monthly price of one token per second**

| Configuration | VRAM | Per month | Tokens/s | $ per token/s |
|---|---|---|---|---|
| [8 × NVIDIA A100 PCIe](https://gpuserver.io/gpu/a100-80gb) GN-A10080×8 | 640 GB | $7,906 | 37 | $213.68 |
| [8 × NVIDIA A100 SXM4](https://gpuserver.io/gpu/a100-sxm4) GN-A100SXM×8 | 640 GB | $8,355 | 39 | $214.23 |
| [4 × NVIDIA H200 SXM5](https://gpuserver.io/gpu/h200) GN-H200×4 | 564 GB | $8,405 | 62 | $135.56 |
| [8 × NVIDIA H100 PCIe](https://gpuserver.io/gpu/h100-pcie) GN-H100PCIE×8 | 640 GB | $10,568 | 38 | $278.11 |
| [8 × NVIDIA H100 SXM5](https://gpuserver.io/gpu/h100-sxm5) GN-H100SXM×8 | 640 GB | $11,509 | 64 | $179.83 |
| [4 × NVIDIA B200 SXM6](https://gpuserver.io/gpu/b200) GN-B200×4 | 720 GB | $11,790 | 103 | $114.47 |
| [8 × NVIDIA H200 SXM5](https://gpuserver.io/gpu/h200) GN-H200×8 | 1128 GB | $15,945 | 91 | $175.22 |
| [8 × NVIDIA B200 SXM6](https://gpuserver.io/gpu/b200) GN-B200×8 | 1440 GB | $22,351 | 152 | $147.05 |

The green cell is the best value on the list, and it is not the cheapest row. Going from $7,906 to $11,790 multiplies the invoice by 1.5× and the throughput by 2.8× — you pay more and each token costs you less. Sorting a catalogue by price and taking the first machine that fits is how a serving budget gets spent twice.

One caveat on that column, and it applies to every price-per-throughput table you will ever read: it uses single-stream generation. Under continuous batching the aggregate climbs several times higher on every row, and it does not climb by the same multiple on each — a node with more spare memory after the weights holds more concurrent sequences. The ranking is stable; the absolute figures are conservative. [The economics of that aggregate](https://gpuserver.io/guides/api-vs-self-hosting) are worked out separately.

## When a mixture of experts is the right answer

In order. Stop at the first line that describes you.

1. **A dense model that fits on one card does the job.** Then take it, and do not look back. One card means no sharding, no interconnect question, no expert-routing traffic, and a machine that costs a fraction of a node. Most products that think they need a frontier open model need a good 32B one.
2. **You need the quality and your context is long.** This is where a mixture of experts is genuinely the best instrument available: you pay once for the weights, and the small cache means the machine keeps serving long conversations without falling over. Size the node on the weights, then check the cache table for your real context.
3. **You need the quality and you serve many people at once.** Same answer, same reason, and the advantage grows with concurrency. The fixed cost is amortised over every stream while the per-stream cost stays low.
4. **You need the quality but only occasionally.** Then you are renting a large node to keep it warm for a few hours a week, which is the worst possible use of a monthly term. Either batch the work so it runs in one sitting, or use a hosted endpoint for this one and keep your own machine for the steady load.
5. **You want to fine-tune one.** Different problem, larger machine: training touches every expert and needs optimiser state for all of them, so the memory that made inference awkward becomes prohibitive. [Adapter fine-tuning](https://gpuserver.io/guides/lora-finetune) on a dense model is the realistic route for almost everyone.

Whichever line you stopped on, the number to check first is the total parameter count, in the precision you intend to serve. Everything else on this page — the speed, the cache, the price per token per second — only becomes relevant once the model is resident. The [configurator](https://gpuserver.io/configure) runs the whole check against every node we rent, and it applies the same routing correction to the throughput estimate as the tables above. If you would rather see the memory arithmetic in full first, [it is a guide of its own](https://gpuserver.io/guides/vram-sizing); if you have not decided on a number format yet, [that choice comes before this one](https://gpuserver.io/guides/awq-vs-gptq-vs-fp8).

## Check your own mixture of experts against real machines.

The configurator holds every node we rent, applies the same arithmetic as this page, and tells you which ones hold your model before you pay for anything.

[Open the configurator](https://gpuserver.io/configure) [Read the guides](https://gpuserver.io/guides)

## Other guides

- [Cost · 6 min When monthly rental beats per-hour The break-even worked out on real numbers, the three costs an hourly price hides until the invoice, and the idle-time trap that triples an estimate. Read the guide](https://gpuserver.io/guides/monthly-vs-hourly)
- [Cost · 9 min Self-hosting against a per-token API The break-even between paying an API per token and renting a GPU, worked out on our own prices and throughput — and the four things the arithmetic leaves out. Read the guide](https://gpuserver.io/guides/api-vs-self-hosting)
- [Practice · 11 min Serving Llama 3.3 70B on one node From a delivered machine to an OpenAI-compatible endpoint, with the flags that matter and the two that silently halve your throughput. Read the guide](https://gpuserver.io/guides/serve-llama-70b)

---

Source: https://gpuserver.io/guides/mixture-of-experts/. This file is generated from the same data as the website; if a figure here differs from a page, the page is authoritative and this file is stale — the canonical source is https://gpuserver.io/.