Practice guide · 10 min read

# Fine-tuning a 70B model on one card.

QLoRA turns a training job that needs a rack into one that needs a single 48 GB GPU. Here is what fits, what it costs for a month, and where the technique stops working.

The short answer

- **QLoRA on 70B:** ~61 GB — fits on one 48 GB card at short sequence lengths, comfortably on an 80 GB one
- **Full fine-tuning of 70B:** ~1,232 GB — a multi-node cluster, which we do not rent
- **The ratio:** Roughly **20×** less memory, for a technique that reaches most of the quality on most tasks
- **What it costs here:** From $286/month, with the card yours for the whole month rather than the run

## What actually fits

LoRA freezes the model and trains a small pair of low-rank matrices beside each weight matrix. QLoRA goes further and holds the frozen weights in 4-bit. What you must keep in memory changes completely as a result.

Frozen weights 4 bits per parameter 35 GB for a 70B model. Read every step, never updated, so they never need a gradient.

Adapters Around 1–2% of parameters A gigabyte or two in BF16. These are the only weights that change, and the only thing you keep at the end.

Optimiser state Adam, on the adapters only Two moments per trained parameter. This is the term that dominates full fine-tuning and nearly vanishes here.

Activations The variable one Scales with batch size and sequence length. Gradient checkpointing trades roughly 30% more compute for a large reduction — the usual lever when a run will not fit.

## Why full fine-tuning does not

Full fine-tuning in BF16 needs about 16 bytes per parameter: 2 for the weight, 2 for its gradient, and 8 for Adam's two moments in FP32, plus the master copy. That is a fixed multiplier, and it is brutal.

**Training memory, QLoRA compared with full fine-tuning**

| Model | QLoRA | Full fine-tune | Ratio | What that means |
|---|---|---|---|---|
| Llama 3.1 8B 8B parameters | 13 GB | 141 GB | 11× | One 24 GB card Full: an 8-GPU node |
| Gemma 3 27B 27B parameters | 27 GB | 475 GB | 17× | One 48 GB card Full: an 8-GPU node |
| Qwen 3 32B 32B parameters | 31 GB | 563 GB | 18× | One 48 GB card Full: an 8-GPU node |
| Llama 3.3 70B 70B parameters | 61 GB | 1,232 GB | 20× | One 80 GB card Full: more than one node |

These are order-of-magnitude figures with a modest allowance for activations — enough to choose a machine, not enough to promise a specific run. Sequence length and batch size move the QLoRA column considerably; the full fine-tuning column barely moves, because it is dominated by the fixed 16 bytes per parameter.

## Model by model

The cheapest node in our catalogue that can train each model with QLoRA, alongside the machine you would need to serve the result.

**Cheapest node for QLoRA training, and for serving the result**

| Model | Training memory | Cheapest node to train on | Cheapest node to serve on |
|---|---|---|---|
| Llama 3.1 8B | 13 GB | [NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) 24 GB per card · $125/mo | [NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $125/mo |
| Mistral Small 24B | 25 GB | [NVIDIA RTX A6000](https://gpuserver.io/gpu/rtx-a6000) 48 GB per card · $286/mo | [NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $125/mo |
| Gemma 3 27B | 27 GB | [NVIDIA RTX A6000](https://gpuserver.io/gpu/rtx-a6000) 48 GB per card · $286/mo | [NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $125/mo |
| Qwen 3 32B | 31 GB | [NVIDIA RTX A6000](https://gpuserver.io/gpu/rtx-a6000) 48 GB per card · $286/mo | [NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $125/mo |
| Llama 3.3 70B | 61 GB | [NVIDIA A100 PCIe](https://gpuserver.io/gpu/a100-80gb) 80 GB per card · $1,091/mo | [2 × NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) $285/mo |

Training wants memory; serving wants memory *and* bandwidth. It is common for the right training card to be a cheap large-memory one and the right serving card to be an expensive fast one — and since both are monthly here, running the two on different machines is often cheaper than compromising on one.

## Running it

Axolotl is preinstalled as one of the images in the configurator. This is a complete run on a single card, from an empty machine.

A QLoRA run on one card

```
$ mkdir -p /scratch/ft && cd /scratch/ft
$ cat > qlora.yml <<'YAML'
base_model: meta-llama/Llama-3.3-70B-Instruct
load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true

sequence_len: 2048
sample_packing: true
gradient_checkpointing: true
micro_batch_size: 1
gradient_accumulation_steps: 16
num_epochs: 2
learning_rate: 0.0001
optimizer: paged_adamw_8bit
bf16: true

datasets:
  - path: /scratch/ft/data.jsonl
    type: chat_template
output_dir: /scratch/ft/out
YAML

$ docker run --rm --gpus all --ipc=host --shm-size=16g \
    -v /scratch/ft:/workspace -v /scratch/models:/root/.cache/huggingface \
    -e HF_TOKEN="$HF_TOKEN" \
    axolotlai/axolotl:main-latest \
    axolotl train /workspace/qlora.yml
```

Watch the memory, not the loss, for the first two minutes

```
$ nvidia-smi dmon -s um

# If memory sits just under the card's total, you are one long batch
# away from an out-of-memory error four hours into the run.
# Lower sequence_len or micro_batch_size now, not then.
```

## The settings that matter

lora_r 16 to 64 The rank, and the main quality-versus-memory dial. 16 is a sensible default; going above 64 rarely pays for itself unless you are teaching genuinely new knowledge rather than a format or a style.

lora_target_linear true Adapts every linear layer rather than attention only. Costs little and consistently outperforms the attention-only default.

sequence_len The single biggest memory lever Activations scale with it. Halving it roughly halves the variable part of your memory. Set it to the length your data actually has, not to the model's maximum.

gradient_checkpointing true, almost always Around 30% slower, and it is what makes a 70B run fit on one card at all. Turn it off only when you have memory to spare.

micro_batch_size & accumulation Trade one against the other The effective batch is their product. Keep micro-batch at 1 and raise accumulation when memory is tight — same gradient, less peak memory, slightly slower.

optimizer paged_adamw_8bit Eight-bit optimiser states with CPU paging as a safety valve. It absorbs the spikes that would otherwise end a run at hour four.

## What a month costs

The reason monthly matters more for training than for serving: a fine-tuning project is not one run. It is a first run that fails on a data bug, a second that overfits, a third that works, and then five more with different ranks.

**Cost of a fine-tuning project, by machine**

| Machine | Card memory | Per month | Suits |
|---|---|---|---|
| [NVIDIA L4](https://gpuserver.io/gpu/nvidia-l4) 300 GB/s | 24 GB | $125/mo | QLoRA up to 8B |
| [NVIDIA RTX 5090](https://gpuserver.io/gpu/rtx-5090) 1,792 GB/s | 32 GB | $335/mo | QLoRA up to 32B |
| [NVIDIA A100 PCIe](https://gpuserver.io/gpu/a100-40gb) 1,555 GB/s | 40 GB | $392/mo | QLoRA up to 32B |
| [NVIDIA RTX A6000](https://gpuserver.io/gpu/rtx-a6000) 768 GB/s | 48 GB | $286/mo | QLoRA up to 32B |
| [NVIDIA A100 PCIe](https://gpuserver.io/gpu/a100-80gb) 1,935 GB/s | 80 GB | $1,091/mo | QLoRA up to 70B |

Compare that with per-hour pricing over a project rather than a run: the [break-even guide](https://gpuserver.io/guides/monthly-vs-hourly) works the arithmetic. A machine that exists for three weeks while you iterate is exactly the case where a meter costs more than a term.

## Where LoRA stops working

It is a very good technique with a real boundary, and it is worth knowing which side of it you are on before renting anything.

**Teaching a genuinely new domain.** LoRA adapts behaviour well and adds knowledge poorly. A model that has never seen your field will not learn it from a low-rank adapter.

**Changing the vocabulary or the tokenizer.** New tokens need embeddings trained, which is outside what an adapter touches.

**Continued pre-training on billions of tokens.** That is full fine-tuning, and it needs the machine in the right-hand column of the table above — which is more than one node, and more than we rent.

**Squeezing the last two points of a benchmark.** Full fine-tuning still wins at the margin. If that margin is what you are being paid for, LoRA is the wrong tool.

For everything else — a tone, a format, a schema, a domain the base model already half knows — QLoRA on a single card reaches most of the quality for a fraction of the machine. That is the trade, and on the models most people actually fine-tune it is a good one.

## Pick the card, keep it for the whole project.

Every configuration is monthly, with root access and no meter — which is what iterating on a fine-tune actually needs.

[Browse the catalogue](https://gpuserver.io/#catalog) [Read the guides](https://gpuserver.io/guides)

## Other guides

- [Payment · 8 min Paying for a server in crypto What actually happens between clicking pay and getting root, which coin to choose, and the four mistakes that lose money on a first payment. Read the guide](https://gpuserver.io/guides/pay-in-crypto)
- [Sizing · 9 min How much VRAM a model actually needs The arithmetic behind “will it fit”: weights, KV cache, and the two places a rule of thumb goes wrong by a factor of three. Read the guide](https://gpuserver.io/guides/vram-sizing)
- [Sizing · 7 min NVLink or PCIe: which one your job needs When the link between cards decides your throughput, when it changes nothing, and how to tell which case you are in before you pay for the wrong node. Read the guide](https://gpuserver.io/guides/nvlink-vs-pcie)

---

Source: https://gpuserver.io/guides/lora-finetune/. This file is generated from the same data as the website; if a figure here differs from a page, the page is authoritative and this file is stale — the canonical source is https://gpuserver.io/.