Infrastructure Checklist for an AI Workload

Before anything runs

1. Request GPU quota. New accounts frequently have a limit of zero, and increases take time. Ask before you need it.
2. Choose the region on GPU availability, not on habit. The card you want may not exist where you usually work.
3. Set a budget with daily alerts. GPU spend accumulates faster than any other resource.
4. Decide where data lives and confirm it satisfies any residency requirement.
5. Configure automatic shutdown for idle instances and notebooks.
6. Establish credentials properly — workload identity, not stored keys.

The quota and the budget alert are the two that most often cause a bad week when skipped.

Data

• Store datasets in object storage, versioned, with lifecycle rules for old versions.
• Keep data in the same region as the compute; cross-region reads are slow and billed.
• Cache to local NVMe on the training instance rather than reading from object storage every epoch.
• Record dataset versions alongside experiment results, or results are not reproducible.
• Anonymise or exclude personal data unless there is a documented basis for using it.
• Track dataset lineage — where it came from and what licence applies.

Reproducible environments

A model that only trains on one person’s machine is not a deliverable.

1. Use a container image with pinned framework, CUDA and driver versions.
2. Pin every dependency with a lockfile.
3. Record the random seed, and accept that full determinism on GPU is often unattainable.
4. Version the training code alongside the configuration that produced each run.
5. Store checkpoints in object storage with the commit and config that created them.
6. Log hyperparameters and metrics to an experiment tracker, not to a spreadsheet.

Training runs

• Checkpoint frequently, to durable storage, so a spot interruption costs minutes rather than hours.
• Make runs resumable from a checkpoint without manual intervention.
• Profile the data pipeline before scaling up; GPU starvation is the usual inefficiency.
• Use early stopping so a run that has stopped improving also stops billing.
• Set a maximum wall-clock time as a safety net.
• Log to a persistent store, not only to the instance.

Serving

1. Separate the serving environment from the training environment entirely.
2. Use an optimised inference runtime rather than the training framework.
3. Set concurrency and rate limits so a traffic spike does not become an unbounded bill.
4. Add a cache in front for repeated inputs.
5. Monitor latency percentiles, error rate and cost per request.
6. Keep the previous model version deployable, so rollback is possible.
7. Version model artefacts explicitly; “the current model” is not a version.

Governance

Worth establishing early rather than retrofitting:

• Which data may be used for training, and on what basis.
• Whether outputs are logged, and whether those logs contain personal data.
• Who approves a model going to production.
• How model performance is monitored for degradation over time.
• What the fallback is when the model or its serving infrastructure is unavailable.
• Which third-party services see your data, and under what terms.

Common questions

Where to go next

See GPU instances and cloud service quotas explained.