Lesson 3 · 10 min
Cloud GPUs — picking the right machine
GPU choice is 50% of cost. Pick wrong and you waste money. Pick righter and you save serious cash.
The rough hierarchy (2026)
| GPU | Memory | Best for | Hourly (ballpark) |
|---|---|---|---|
| L40S | 48GB | Inference, mid models | $1-1.5/hr |
| A100 40GB | 40GB | Inference, training | $1.5-2.5/hr |
| A100 80GB | 80GB | Larger inference, training | $2-3.5/hr |
| H100 80GB | 80GB | Frontier training, lowest-latency inference | $3-7/hr |
| H200 / B100 | 96-141GB | Latest-gen training, very large models | $5-10/hr |
| AMD MI300X | 192GB | Big-memory inference, less software support | $2-4/hr |
| TPU v5e/v5p | varies | Google Cloud, JAX/PyTorch-XLA | $1-3/hr |
Hourly prices vary wildly by provider, region, and reservation type.