Fine-tuning a 70B model on one card.

What actually fits

LoRA freezes the model and trains a small pair of low-rank matrices beside each weight matrix. QLoRA goes further and holds the frozen weights in 4-bit. What you must keep in memory changes completely as a result.

Why full fine-tuning does not

Full fine-tuning in BF16 needs about 16 bytes per parameter: 2 for the weight, 2 for its gradient, and 8 for Adam's two moments in FP32, plus the master copy. That is a fixed multiplier, and it is brutal.

These are order-of-magnitude figures with a modest allowance for activations — enough to choose a machine, not enough to promise a specific run. Sequence length and batch size move the QLoRA column considerably; the full fine-tuning column barely moves, because it is dominated by the fixed 16 bytes per parameter.

Model by model

The cheapest node in our catalogue that can train each model with QLoRA, alongside the machine you would need to serve the result.

Training wants memory; serving wants memory and bandwidth. It is common for the right training card to be a cheap large-memory one and the right serving card to be an expensive fast one — and since both are monthly here, running the two on different machines is often cheaper than compromising on one.

Running it

Axolotl is preinstalled as one of the images in the configurator. This is a complete run on a single card, from an empty machine.

$ mkdir -p /scratch/ft && cd /scratch/ft
$ cat > qlora.yml <<'YAML'
base_model: meta-llama/Llama-3.3-70B-Instruct
load_in_4bit: true
adapter: qlora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true

sequence_len: 2048
sample_packing: true
gradient_checkpointing: true
micro_batch_size: 1
gradient_accumulation_steps: 16
num_epochs: 2
learning_rate: 0.0001
optimizer: paged_adamw_8bit
bf16: true

datasets:
  - path: /scratch/ft/data.jsonl
    type: chat_template
output_dir: /scratch/ft/out
YAML

$ docker run --rm --gpus all --ipc=host --shm-size=16g \
    -v /scratch/ft:/workspace -v /scratch/models:/root/.cache/huggingface \
    -e HF_TOKEN="$HF_TOKEN" \
    axolotlai/axolotl:main-latest \
    axolotl train /workspace/qlora.yml

$ nvidia-smi dmon -s um

# If memory sits just under the card's total, you are one long batch
# away from an out-of-memory error four hours into the run.
# Lower sequence_len or micro_batch_size now, not then.

The settings that matter

What a month costs

The reason monthly matters more for training than for serving: a fine-tuning project is not one run. It is a first run that fails on a data bug, a second that overfits, a third that works, and then five more with different ranks.

Compare that with per-hour pricing over a project rather than a run: the break-even guide works the arithmetic. A machine that exists for three weeks while you iterate is exactly the case where a meter costs more than a term.

Where LoRA stops working

It is a very good technique with a real boundary, and it is worth knowing which side of it you are on before renting anything.

For everything else — a tone, a format, a schema, a domain the base model already half knows — QLoRA on a single card reaches most of the quality for a fraction of the machine. That is the trade, and on the models most people actually fine-tune it is a good one.