What a mixture of experts really costs.

The short version

A mixture of experts splits the feed-forward part of each layer into many small networks and sends every token through only a few of them. The published name records both facts: the total number of parameters, and the number that are active for any one token. You must hold all of the first in memory, and you only read the second per token. That single sentence is the whole guide, and getting it backwards is the most expensive mistake we see people make before an order.

The consequence is a model with a strange cost shape: it rents like a giant and runs like a mid-size model. Whether that is a bargain or a trap depends entirely on which of the two numbers your bottleneck is.

The arithmetic below is the same one the configurator runs, on the same catalogue. If you have not yet worked out how much memory a model needs at all, that formula comes first — this page is what changes once the model is a mixture of experts.

Reading the name

Three naming conventions are in circulation and they all encode the same pair of numbers, which is why the confusion survives. Once you can read them, the sizing question answers itself.

The last column is the one to remember. A dense model reads everything it stores; a mixture of experts reads a slice. Everything strange about these models — good and bad — comes out of that one line.

The memory you actually rent

Here is the whole trap in one table. The middle column is what the model needs on the machine; the column after it is what a reader who took the active number at face value would have budgeted for. The last column is the bill for that misreading.

Read the last column as a shopping error. Someone sizing on the active count alone would go looking for a single card and end up needing a whole node — not a slightly bigger machine, a different category of machine, at a different price.

What the active parameters buy

Generation is limited by memory bandwidth: each new token requires re-reading the weights that participate in it. A mixture of experts reads only its active slice, so the ceiling on its speed is set by a much smaller number than its size suggests. That part of the promise is real, and it is why these models exist at all.

The table below puts every model on the same machine — 8 × NVIDIA A100 PCIe at $7,906 a month, the cheapest node in our catalogue that holds all of them at once. Comparing throughput figures taken on different machines compares nothing.

The two rows worth comparing are the largest mixture of experts and the largest dense model, because both are split across the whole node. The mixture is the bigger model by a wide margin and it still generates faster, since it reads a fraction of itself per token — that is the trade the architecture exists to make. Read the rows marked on one card differently: those models occupy a single card of the node and leave the other seven free, which is a cheaper machine waiting to be chosen, not a slower one.

The second thing the arithmetic hides is that a mixture of experts is harder to shard well. A dense model split across cards synchronises once per layer; a mixture also has to route tokens to whichever card holds the chosen expert, which is a different and less predictable traffic pattern. This is one of the few workloads where the interconnect genuinely earns its price rather than merely appearing on the invoice.

The half that gets cheaper

Everything so far has been bad news for the big models. Here is the compensation, and it is a large one that almost no comparison mentions: the models with the most parameters on this page have the least key/value cache per token. The cache is set by the attention design — layers, key/value heads, head dimension — and it has nothing to do with how many experts sit behind attention.

Those columns are per concurrent request. Multiply by the number of people using the endpoint at once and the ranking in the last column decides your machine far more than the weights do.

The practical reading: weights are a fixed admission fee, cache is the running cost. A mixture of experts charges an enormous admission fee and then a small running cost per user. A dense model of comparable ability is the other way round. Which of the two is cheaper for you is decided by concurrency and context length, not by the parameter count on the model card.

Which machines hold them

At 4-bit, with the reference context, and taking the cheapest node in our catalogue that holds each model — whether on one card or split across the node. These are the entry tickets, not recommendations; the next section explains why.

Every one of these is a multi-card node, and that is the honest headline of this page: nothing in our catalogue holds one of these models on a single card, however small the active count makes them look. The full catalogue has the rest of the configurations, and the configurator will run this same check against your own model and context.

The cheapest node is the wrong one

Fitting is a threshold, not a goal. Once a model is split across a node, throughput depends on the memory bandwidth of the cards it is split across — and bandwidth per dollar varies by more than a factor of two across configurations that all clear the same memory bar. Here is DeepSeek V3 671B-A37B (MoE) on every node that holds it, ordered by price, with the only column that matters on the right.

The green cell is the best value on the list, and it is not the cheapest row. Going from $7,906 to $11,790 multiplies the invoice by 1.5× and the throughput by 2.8× — you pay more and each token costs you less. Sorting a catalogue by price and taking the first machine that fits is how a serving budget gets spent twice.

One caveat on that column, and it applies to every price-per-throughput table you will ever read: it uses single-stream generation. Under continuous batching the aggregate climbs several times higher on every row, and it does not climb by the same multiple on each — a node with more spare memory after the weights holds more concurrent sequences. The ranking is stable; the absolute figures are conservative. The economics of that aggregate are worked out separately.

When a mixture of experts is the right answer

In order. Stop at the first line that describes you.

1. A dense model that fits on one card does the job. Then take it, and do not look back. One card means no sharding, no interconnect question, no expert-routing traffic, and a machine that costs a fraction of a node. Most products that think they need a frontier open model need a good 32B one.
2. You need the quality and your context is long. This is where a mixture of experts is genuinely the best instrument available: you pay once for the weights, and the small cache means the machine keeps serving long conversations without falling over. Size the node on the weights, then check the cache table for your real context.
3. You need the quality and you serve many people at once. Same answer, same reason, and the advantage grows with concurrency. The fixed cost is amortised over every stream while the per-stream cost stays low.
4. You need the quality but only occasionally. Then you are renting a large node to keep it warm for a few hours a week, which is the worst possible use of a monthly term. Either batch the work so it runs in one sitting, or use a hosted endpoint for this one and keep your own machine for the steady load.
5. You want to fine-tune one. Different problem, larger machine: training touches every expert and needs optimiser state for all of them, so the memory that made inference awkward becomes prohibitive. Adapter fine-tuning on a dense model is the realistic route for almost everyone.

Whichever line you stopped on, the number to check first is the total parameter count, in the precision you intend to serve. Everything else on this page — the speed, the cache, the price per token per second — only becomes relevant once the model is resident. The configurator runs the whole check against every node we rent, and it applies the same routing correction to the throughput estimate as the tables above. If you would rather see the memory arithmetic in full first, it is a guide of its own; if you have not decided on a number format yet, that choice comes before this one.