Doubling the GPUs on a training job rarely halves the time it takes. The first additions come close, each subsequent one returns less, and at some point the marginal GPU adds cost without moving the completion date enough to matter.
The shape of that decline is a scaling curve, and where it bends decides whether a cluster earns its keep.
GPU optimisation for AI training is usually argued in terms of peak specifications. The financially useful question is where a given configuration puts a workload on that curve, which is a property of the job rather than of the accelerator.
What a scaling curve measures
Scaling efficiency is the ratio between the speed-up obtained and the speed-up that perfect scaling would give. Eight GPUs finishing a job four times faster than one is 50% efficiency, and the missing half has gone into communication overhead and time spent waiting on data.
Plotted against GPU count, that ratio falls into three regions:
- Starved – The job is constrained by insufficient compute or memory, and adding capacity produces close to linear returns.
- Efficient – GPUs stay busy and each addition still buys a meaningful reduction in wall-clock time.
- Saturated – Cost climbs faster than performance, and the marginal accelerator is largely paying to wait.
The knee between the second and third regions is where another unit of compute stops returning enough to justify itself.
It sits in a different place for every combination of model architecture, dataset and deadline, which is why a curve borrowed from a vendor benchmark is of limited use for planning a purchase.
What puts the knee where it is
Several constraints compete to set it, and which one binds changes with the job. Gradient synchronisation volume grows with parameter count, so a larger model spends proportionally more of each step exchanging data rather than computing.
Storage throughput caps how quickly batches reach the accelerators, and a pipeline sized for eight GPUs will starve sixteen.
Memory capacity per GPU determines whether a model can be held whole or has to be partitioned, and partitioning converts a memory problem into a communication problem.
Host-side preprocessing sets a fourth ceiling that is easy to miss, because a CPU-bound data loader looks like a slow GPU from every dashboard.
The fabric carrying that traffic is a decision in its own right, worked through in the comparison of InfiniBand and Ethernet for AI clusters.
Finding your own knee
Run a representative job at one, two, four and eight GPUs, and record wall-clock time to a defined quality target rather than throughput. Throughput rewards configurations that process more batches without converging any sooner.
This is the methodology MLPerf Training uses, measuring the time a system takes to train a model to a specified quality threshold rather than its theoretical compute.
Two details from its rules are worth borrowing. Results are produced by running each benchmark several times, discarding the highest and lowest, and averaging the rest.
And even after that, MLCommons puts residual variance at roughly 2.5% on imaging benchmarks and roughly 5% on the others.
That second figure sets a noise floor for your own testing. A configuration change that appears to buy 3% is not yet distinguishable from run-to-run variation, which matters when a procurement case is being built on a single comparison run.
Turning the curve into a cost
Scaling efficiency becomes a margin question once it is expressed in GPU-hours. Take a job that runs in 100 hours on a single accelerator.
At eight GPUs and 50% efficiency it finishes in 25 hours, having consumed 200 GPU-hours against the 100 the single-GPU run would have used. The result arrives four times sooner and costs twice as much.
Push to sixteen GPUs and suppose efficiency falls to 30%. Wall-clock time improves to just under 21 hours so the job finishes around 17% sooner than the eight-GPU run, while GPU-hours rise to roughly 333, an increase of about 67%.
That is the knee expressed in money: a sixth off the schedule for two-thirds more spend.
Whether the trade is worth making depends on what a day of schedule is worth. A team blocked from testing the next variant may value it highly. A quarterly retraining run with a week of slack does not, and the same cluster should be scheduled differently for each.
What a hardware generation does to the curve
A new generation moves the knee outward, and the useful analysis is which resource it moves it with. Comparing NVIDIA’s eight-GPU systems on a like-for-like basis, the DGX H100 carries 640 GB of GPU memory with 7.2 TB/s of aggregate GPU-to-GPU NVLink bandwidth and draws around 10.2 kW at maximum system power.
The DGX B200 carries 1,440 GB with 14.4 TB/s aggregate and draws around 14.3 kW.
Memory grows by a factor of 2.25, interconnect bandwidth by 2, and maximum system power by roughly 1.4. The resources that relieve scaling constraints are growing faster than the power bill that pays for them, which is a more defensible argument for a generation change than a headline throughput multiple.
That only converts into margin if the constraint being relieved is the one actually binding. A job comfortably inside 80 GB per GPU, running at good efficiency at its current scale, gains little from 180 GB per GPU and still pays the additional power and cooling.
Where memory pressure is forcing partitioning, the same upgrade can remove the communication overhead that partitioning introduced.
The generations have moved on again. Blackwell Ultra puts 2.3 TB of memory in the same eight-GPU footprint at around 14.5 kW, roughly 3.6 times the memory of a DGX H100 for about 1.4 times the power. Which specific part suits a given workload is covered in the GPU selection comparison.
Where does the margin leak?
Curves are drawn from measured performance, and the gap between measured and theoretical is usually occupied by idle silicon.
A cluster can report full allocation while its accelerators spend a large share of each step waiting on storage, on a straggler in the collective, or on a scheduler holding a job for a resource it does not need.
Allocation and productive utilisation are different numbers, and only the second one appears on the curve. An accelerator held at low productive utilisation is stranded capital regardless of what it cost.
Tracking energy consumed per completed run alongside utilisation closes the loop between the curve and the invoice, and the method for that measurement is set out in the post on tokens per watt.
Hardware choice sets the shape of the curve, and operating discipline decides where on it you actually sit. The wider economics of GPU capacity are set out on the GPU compute and cloud pillar.
Neurotechnology Cloud runs training workloads on dedicated European capacity through its AI Cloud and managed GPU services, where compute, storage, networking and orchestration are sized against the workload rather than sold as a fixed configuration.
If you want to establish where your own workloads sit on the curve before committing to capacity, contact us today.
