Serving Llama 3.3 70B on one node.

What you need

70 billion parameters. At 4-bit that is 35 GB of weights; the KV cache adds 2.5 GB at 8k context for a single request, and the runtime overhead about 15%. Everything else follows from those three numbers — the sizing guide shows where they come from.

Note the 4-bit row at 128k: the weights shrink to 35 GB but the cache does not shrink at all, because AWQ and GPTQ quantise weights only. That single fact decides most of the machine choice below.

Choosing the node

The cheapest machines in our catalogue that hold this model on a single card, which is the configuration you want if you can have it — no sharding means no synchronisation and one less thing to debug.

Throughput is single-stream generation, bounded by memory bandwidth, and deliberately conservative. With continuous batching the aggregate across concurrent requests is several times higher — that is the whole point of vLLM.

Ten minutes of setup

Nothing here is specific to us except the address. Docker, the NVIDIA container toolkit and a working driver stack are already on the machine.

$ ssh root@203.0.113.42
$ nvidia-smi --query-gpu=name,memory.total --format=csv

# Weights on the fast local NVMe, not the system volume.
$ mkdir -p /scratch/models
$ df -h /scratch

# A gated model needs a token. Skip if yours is open.
$ export HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxx

Starting the server

One container. The first run downloads roughly 40 GB of quantised weights, which on a 1 Gbit/s port takes a few minutes; every restart afterwards is seconds.

$ docker run -d --name vllm --restart unless-stopped \
    --gpus all --ipc=host \
    -v /scratch/models:/root/.cache/huggingface \
    -e HF_TOKEN="$HF_TOKEN" \
    -p 127.0.0.1:8000:8000 \
    vllm/vllm-openai:latest \
    --model casperhansen/llama-3.3-70b-instruct-awq \
    --quantization awq_marlin \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.92

# Watch it load. "Application startup complete" is the signal.
$ docker logs -f vllm

$ curl -s http://127.0.0.1:8000/v1/models | head

$ curl -s http://127.0.0.1:8000/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"casperhansen/llama-3.3-70b-instruct-awq",
         "messages":[{"role":"user","content":"In one sentence: what is a KV cache?"}],
         "max_tokens":80}'

The flags that matter

Not publishing it to the world

Reach it from your own machine over an SSH tunnel — no port opened, no certificate to manage, nothing to misconfigure:

$ ssh -N -L 8000:127.0.0.1:8000 root@203.0.113.42

# Now http://127.0.0.1:8000 on your laptop is the server.

If it genuinely must be public, put a reverse proxy in front with TLS and an API key, and restrict the source addresses in the firewall as well. The documentation has a minimal ruleset. Belt and braces is the correct posture for an endpoint that will answer anyone who asks.

Getting the throughput up

In the order worth trying. The first two are free and usually give the largest gains.

1. Lower --max-model-len to your real context. Almost always the biggest single win, and it costs nothing you were using.
2. Raise --gpu-memory-utilization to 0.95. This is a dedicated machine; there is nothing else on the card to leave room for.
3. Turn on FP8 KV cache if the card supports it. Doubles what you can hold.
4. Batch on the client side. Continuous batching only helps if requests arrive concurrently. One-at-a-time requests leave most of the card idle whatever you configure.
5. Then, and only then, consider a faster card. Generation is bounded by memory bandwidth, so an H200 at 4,800 GB/s is roughly 2.4× an H100 PCIe at 2,000 for the same model — a real gain, and the most expensive one on this list.

$ docker exec vllm python -m vllm.entrypoints.openai.api_server --help | head -1
$ docker exec vllm vllm bench serve \
    --model casperhansen/llama-3.3-70b-instruct-awq \
    --num-prompts 200 --request-rate 8

# And watch the card while it runs: if utilisation sits below 90%,
# the bottleneck is your client, not the GPU.
$ nvidia-smi dmon -s um

Keeping it running

Three things worth doing on the first day, because all three are annoying to discover on the fourteenth.

And when the term ends, the disks are erased within the hour with no grace period. Everything on /scratch goes with it. That is deliberate — it is the same promise that guarantees the machine you were given had nobody else's data on it — but it means the last day of a term is not the day to start copying things off.