Cloud GPUs for AI Workloads: What to Look For
How to choose a cloud GPU for training, fine tuning, or inference, what Vultr Cloud GPU, EC2 GPU families, Google Cloud accelerators, and OCI GPU shapes offer, and how quotas and cost control work.
Renting a GPU in the cloud is the fastest way to train a model, fine tune an open weights model on your own data, or serve inference without buying hardware that will be outdated in two years. The catch is that GPU instances are expensive per hour, scarce in many regions, and gated by quotas that start low on a new account. This guide explains how to match a GPU to the job, what Vultr, AWS, Google Cloud, and Oracle Cloud actually offer, how quotas and availability work, and the operating habits that stop a GPU bill from surprising you.
Match the GPU to the workload
There are three broad jobs. Training from scratch needs the largest memory, the fastest interconnect between GPUs, and usually several GPUs on one host or across hosts; this is the territory of data centre cards such as the NVIDIA A100 and H100 family. Fine tuning an existing model with methods such as LoRA needs far less: a single mid range or high memory card is often enough, and the job finishes in hours rather than weeks. Inference, meaning serving a trained model to users, is limited by memory and latency rather than raw compute, so cards designed for inference such as the NVIDIA L4, L40S, or A10 give better value than a training card that sits mostly idle.
Before you rent anything, work out the memory the model needs. A model’s weights must fit in GPU memory along with the activations and, for training, the optimizer state; quantised weights and smaller batch sizes reduce the requirement. If the model does not fit on one card you need multi GPU shapes and a framework that can shard it, which raises both the cost and the engineering effort sharply.
What to check on a spec sheet
- GPU memory per card decides which models fit at all. It matters more than the card’s generation for most inference work.
- Interconnect between GPUs on the same host (NVLink on high end NVIDIA shapes) and between hosts (high bandwidth networking on cluster shapes) matters for multi GPU training and barely at all for single card inference.
- Fractional GPUs, offered by Vultr and others as vGPU plans, give a slice of a card for development and light inference at a lower price than a whole card.
- Local NVMe storage on the instance keeps data loading from becoming the bottleneck; a training job that reads from slow network storage wastes GPU hours.
- CPU and system memory still matter for data preprocessing and for loading model weights before they reach the GPU.
- Driver and image support: a provider image with the NVIDIA driver, CUDA, and container toolkit preinstalled saves the first afternoon.
What Vultr, AWS, Google Cloud, and Oracle Cloud offer
Vultr Cloud GPU sells whole and fractional NVIDIA GPUs as cloud instances and bare metal, with a lineup that has included the A16, A40, L40S, A100, H100, and newer accelerators, plus AMD Instinct options, across a subset of its locations. Its appeal is simplicity: an hourly instance with a GPU attached, created from the same control panel and API as an ordinary server.
On AWS, GPU capacity lives in EC2 instance families. The P family (P4 with A100, P5 with H100 and later cards) targets distributed training with high bandwidth networking. The G family (G5 with A10G, G6 with L4, G6e with L40S) targets inference, graphics, and smaller training jobs. AWS also offers its own Inferentia and Trainium chips through the Inf and Trn families, and the Deep Learning AMIs ship with drivers and frameworks preinstalled. EC2 Capacity Blocks for ML let you reserve GPU capacity for a defined window when on demand availability is tight.
Google Cloud attaches GPUs to Compute Engine through accelerator optimised machine families: A2 with A100, A3 with H100 and later, and G2 with L4 for inference. Google also offers TPUs, its own accelerators, which suit frameworks such as JAX and PyTorch with XLA. Deep Learning VM images and Vertex AI managed training remove much of the setup for teams that would rather not manage instances.
Oracle Cloud Infrastructure names GPU capacity as shapes, such as VM.GPU.A10 virtual machines for inference and bare metal shapes such as BM.GPU.H100.8 and BM.GPU.A100 for training, with RDMA cluster networking available on the bare metal tiers for multi node work.
| Provider | How GPUs are offered | Inference oriented options | Training oriented options |
|---|---|---|---|
| Vultr | Cloud GPU instances and bare metal, whole or fractional | Fractional vGPU plans, A16, L40S | A100, H100, and newer whole GPU and multi GPU servers |
| AWS | EC2 instance families, on demand, spot, or Capacity Blocks | G5, G6, G6e, Inf2 | P4, P5 and later, Trn1 |
| Google Cloud | Accelerator optimised Compute Engine families and TPUs | G2 with L4 | A2, A3, TPU pods |
| Oracle Cloud | GPU shapes, virtual machine and bare metal | VM.GPU.A10 shapes | BM.GPU.A100 and BM.GPU.H100 bare metal with cluster networking |
Exact card lists, regions, and prices change often on every provider, so treat the table as a map of the families rather than a current catalogue, and confirm details on the provider’s own pages before committing.
Quotas, availability, and reservations
A new account on any hyperscaler usually starts with a GPU quota of zero in every region. AWS enforces this as a vCPU quota for the G and P instance families, Google Cloud as a per region GPU quota, and Oracle Cloud through service limits on each GPU shape. Increasing the quota means submitting a request through the console’s quota or limits page, stating the instance type, region, and intended use, and waiting for review; providers approve increases based on account history and the reason given. Plan for that lead time and follow the provider’s process rather than looking for shortcuts, which is the fastest way to lose an account.
Quota is permission, not capacity. A region can be out of the card you want even with quota approved, especially for the newest generations. Design so the job can run in more than one region, check availability before scheduling a long run, and use reservations such as Capacity Blocks where a deadline makes availability non negotiable. The Cloud Library guide to cloud service quotas explains the request process in more depth.
Operating habits that control cost
The GPU meter runs whether or not the card is busy. Stop or destroy instances the moment a job finishes, and automate it: a training script that shuts the instance down on completion pays for itself the first time someone forgets. Checkpoint to object storage at regular intervals so an interrupted run resumes rather than restarts, which also makes spot and preemptible GPU capacity usable at a discount for tolerant workloads. Build a machine image with drivers and dependencies once, then launch from it, so every new instance is productive within minutes. Set a budget alert with a threshold well below the expected monthly spend, and put GPU work in its own project or account so its cost is visible on its own line.
Where to go next
GPU work needs an account with the right family available and a quota request behind it. The marketplace lists Vultr accounts with credit variants that can go toward Cloud GPU instances, AWS accounts for EC2 GPU families, Google Cloud accounts for accelerator optimised machines and TPUs, and Oracle Cloud accounts for GPU shapes, each with the variant labels stated on the page.
If you would rather call a hosted model than run your own GPU, read getting started with Amazon Bedrock. For the wider Vultr platform, see Vultr explained. The Cloud Library hub lists every developer workflow guide.