Jensen Huang made tokens per watt a boardroom term at GTC 2026, arguing that a power-constrained data centre should be judged on how much intelligence it manufactures from its electrical envelope rather than on what its accelerators can theoretically do.
The framing stuck because the constraint is real. Where grid interconnections in some markets take five to seven years, a facility capped at 100 MW cannot buy its way past the ceiling.
The metric spread faster than any agreed method of calculating it. Our guide to GPU cloud computing covers where it sits among wider cost dynamics, and our piece on tokenomics and the compute economy covers how hardware generations have moved it.
This post covers the mechanics: how the number is produced, where the measurement boundary sits, and how to interrogate a figure somebody else has published.
What the number actually is
Tokens per watt is output-token throughput divided by average system power:
Tokens per watt = output tokens per second ÷ average system power in watts
A watt is a joule per second, so the units cancel to tokens per joule. The metric describes energy converted into output, which is why it maps onto a facility’s power envelope rather than its capital budget.
Two systems built on the same accelerator can land far apart, because model architecture, numerical precision, inference runtime and batching policy all move the result.
How it relates to PUE
Data centre operators already have an efficiency metric, and the two answer different questions.
PUE compares total facility energy against the energy reaching IT equipment, so it grades the building: cooling, power distribution, everything outside the compute itself. A facility can post an excellent PUE while the servers inside it convert that clean, efficiently delivered power into very little useful work.
Tokens per watt grades the compute layer instead, measuring productivity rather than distribution. The two are complementary, and an infrastructure case built on one without the other has a blind spot. Strong PUE with weak tokens per watt means efficient delivery of power to inefficient work.
Why throughput alone misleads
A system generating more tokens per second may draw disproportionately more power, and where that power cannot be supplied economically the headline figure stops being useful.
| System | Output throughput | Average power | Tokens per watt |
| Platform A | 6,000 tokens/second | 1,000 W | 6 |
| Platform B | 8,000 tokens/second | 800 W | 10 |
Platform B is 33% faster on throughput. The efficiency gap is wider: roughly 67% more tokens from the same power budget, or 40% less energy per token.
Operating point matters as much as hardware here. NVIDIA claims that running GPUs at the most efficient point on their power-performance curve can support up to 40% more accelerators inside an unchanged power budget, which for a 100 MW site is worth the equivalent of 40 MW without a new interconnection.
The measurement boundary decides the answer
This is where most published comparisons quietly fall apart. GPU draw is one component, and inference also consumes CPU, memory, storage, networking and cooling. A benchmark measuring accelerator power and one measuring a full rack are not describing the same system.
The MLCommons Power working group resolves this by measuring AC power at the wall for the entire system, using SPEC PTDaemon and a certified analyser, over a minimum sixty-second window. MLCommons states plainly that its methodology validates full system power only, and that TDP configurations and power supply ratings are neither measured nor validated by them.
That is worth carrying into vendor conversations. A tokens-per-watt figure derived from a published TDP rests on a number the standards body declines to treat as measured.
Duration deserves the same scrutiny. Sixty seconds is a floor for a benchmark submission, not a proxy for production. An accelerator holding throughput through a sustained inference load scores very differently from one throttling once the rack reaches thermal equilibrium, and short measurement windows systematically flatter the second kind.
For a like-for-like comparison, hold constant:
- The model and model version
- Numerical precision and quantisation settings
- Prompt and output lengths
- Runtime, drivers and inference framework
- Concurrency and batching policy
- Measurement duration and thermal state
- The boundary itself, whether accelerator, server or rack
Useful tokens, and what accuracy costs
A platform can look efficient because it generates longer answers, batches aggressively or accepts slower responses. None of that necessarily creates business value, so the figure worth tracking is useful tokens per watt: output meeting the quality and latency thresholds the use case requires.
MLPerf Power results put a number on the trade-off. Earlier benchmark rounds showed organisations giving up as much as 50% of their energy efficiency to lift inference accuracy from 99% to 99.9%.
Reduced-precision techniques such as quantisation have since narrowed that gap considerably, which is the point: the accuracy target and the efficiency figure move together, and a comparison that does not state its accuracy threshold has not said enough to act on.
Turning it into money
Tokens per watt becomes a budget input once converted into energy per million tokens:
kWh per million tokens = 1 ÷ (3.6 × tokens per watt)
Multiply by the effective electricity cost, then add cooling, depreciation, networking, software, support and any reserved capacity sitting idle.
Energy is rarely the largest line item, which is why the metric earns its place for a second reason. It is a proxy for infrastructure density, so a more efficient system produces more output inside the same electrical and cooling footprint and can defer a facility upgrade. The deferred capital expenditure frequently exceeds the energy saving outright.
Reading someone else’s number
Independent measurement and vendor projection circulate under identical vocabulary, and the difference decides how much weight a procurement case can carry.
SemiAnalysis InferenceMax benchmarks, which are third-party, put NVIDIA’s GB300 NVL72 at roughly 50 times the performance per watt of H200 with around 35 times lower cost per token.
NVIDIA’s own Vera Rubin projections claim up to 10 times more tokens per megawatt than GB200 NVL72, rising to 35 times when paired with the Groq 3 LPX rack. Those are self-reported for a platform still in production ramp.
The baseline is where most of the ambiguity lives, and the field’s most-quoted figure shows why. Huang’s headline for a gigawatt facility, around 700 million tokens per second on Vera Rubin, gets reported against whichever comparison the writer picked up: roughly 2 million for a Hopper-era x86 system, or around 22 million for the Blackwell generation.
Both are defensible. They produce very different multiples, about 350x against the first and closer to 32x against the second, and the larger number travels further precisely because it is larger. A team currently running Blackwell reading the 350x figure is looking at a gain measured from hardware two generations behind what they own.
Three questions resolve most of this:
- Against what baseline, named by exact configuration rather than architecture family?
- Measured or projected, and by whom?
- On which model, at what concurrency and what accuracy threshold?
Dedicated environments make these answerable, because the models and concurrency patterns under test are your own rather than a vendor reference configuration. Neurotechnology Cloud provides AI cloud and managed GPU services on dedicated or shared environments where that benchmarking runs against real workloads.
The question for the next review
Efficiency figures collapse the moment the surrounding stack becomes the constraint. A starved data path leaves accelerators waiting on storage or fabric and tokens per watt records the stall as inefficiency without identifying its cause.
So the question worth putting to a vendor is narrower than which platform produces the most tokens per second. Ask how many tokens meeting your accuracy and latency thresholds it sustains per watt, at a stated boundary, over a realistic duration, on your model at your concurrency. Anything less specific is describing somebody else’s workload.
