Training vs Inference: Two Very Different Cost Problems

The shapes

This is why spot capacity suits training and not user-facing inference, and why the same card can be right for one and wrong for the other.

Reducing training cost

1. Use spot or preemptible instances with frequent checkpointing. The largest single saving available.
2. Fine-tune rather than train from scratch. Parameter-efficient methods can reduce cost by orders of magnitude.
3. Use mixed precision. Lower precision training is faster and uses less memory, usually with no meaningful quality loss.
4. Fix the data pipeline first. A GPU waiting on data is the most expensive idle resource you own.
5. Run small experiments small. Validate on a subset before committing to a full run.
6. Stop early. Automatic early stopping prevents paying for epochs that add nothing.
7. Choose the region on price. GPU pricing varies meaningfully between regions.

Reducing inference cost

1. Batch requests. Processing several together dramatically improves throughput per unit of hardware.
2. Quantise the model. Lower precision usually reduces memory and increases speed with acceptable quality.
3. Use an optimised serving runtime rather than the training framework.
4. Cache results for repeated identical inputs. Frequently the largest saving, and often overlooked.
5. Right-size the hardware. Many models serve acceptably on CPU at modest volume.
6. Autoscale on queue depth, and scale to zero if latency requirements allow.
7. Consider a smaller model. A distilled model that is nearly as good and much cheaper is often the correct engineering answer.

Caching deserves emphasis. In many applications a substantial share of requests are near-duplicates, and serving those from a cache costs almost nothing.

The build-or-call decision

Self-hosting a model means paying for capacity. Calling a hosted API means paying per token or per request. The crossover depends on volume and utilisation, not on principle.

• Low or spiky volume — a hosted API is almost always cheaper, because you pay nothing when idle.
• High steady volume — self-hosting becomes cheaper once utilisation is high enough to justify always-on hardware.
• Data residency or privacy requirements — may make self-hosting necessary regardless of cost.
• Specialised or fine-tuned models — self-hosting, unless the provider supports your customisation.

Calculate it: hourly instance cost divided by requests per hour gives cost per request. Compare that against the API price at your volume, and include the engineering time to operate the serving stack.

Costs that are easy to miss

• Storage for datasets, checkpoints and model artefacts, which accumulates quickly.
• Egress when moving datasets between regions or providers.
• Idle GPU time between experiments — often a large share of total GPU spend.
• Development and experimentation notebooks left running.
• Vector database capacity, which grows with your corpus.
• Failed runs, which cost exactly as much as successful ones.

Tag every machine learning resource by experiment or project, or attributing this spend afterwards becomes impossible.

Common questions

Where to go next

See GPU instances and cutting a cloud bill.