Ask an AI a question. Most of what the chip spends to answer it isn’t math.
That sounds like a slogan, so here’s the measurement. One model and one setting throughout: Qwen3 1.7B in its published FP8 checkpoint, on an NVIDIA B300, with a 16K-token conversation, decoding only.
Batching trades your time for energy
A GPU serving a small model has one dial: how many people share it.
One person gets a token every 2.2 ms, at 0.99 J a token. Sixteen people each get a token every 6.4 ms, at 0.35 J.
Batching cut each token’s energy to about a third. It also made all sixteen wait three times longer for every word. Total output went up five times, from 445 tokens a second to 2,346, which is why the dial exists. The operator’s saving was paid in full by the people waiting.
Figure 1. The batching dial
Qwen3 1.7B on one NVIDIA B300, a 16K-token conversation. Each point is one setting of the dial, from one person sharing the GPU to sixteen: energy for each token against the time between a person’s tokens.
Measured on an NVIDIA B300, decoding only
The dial only moves the bill. Turn it one way and the people waiting pay in time; turn it the other and the operator pays in energy. No setting is good at both.
What the GPU is waiting on
To write a token, the GPU reads the whole model in from memory, and then everything the person has said so far: the keys and values the model keeps for every earlier token of the conversation. Both are easy to size from the model’s public configuration. The model is about 2 GB, and a 16K-token conversation is about 1.8 GB. With one person, every token reads both, about 3.8 GB.
At the B300’s 8 TB/s, 3.8 GB takes under half a millisecond. The token takes 2.2 ms. So for one person, the reads aren’t what the GPU waits on. Most of the other 1.7 ms is overhead of the kind part 1 found, a fixed cost for every token whatever the model’s size, and the board draws 461 W through all of it.
Batching shares that fixed cost, and the model’s read, among everyone in the step. That’s why the energy falls. It can’t share the conversations. Each one is read for its own person, so with sixteen people every step reads about 31 GB, nearly all of it conversation, and reading it takes most of the step’s 6.4 ms. Everyone waits.
Figure 2. What each token reads
Qwen3 1.7B with a 16K-token conversation: gigabytes read from memory for each person’s token, by people at once.
Calculated from Qwen3 1.7B’s public configuration, not measured
Long conversations make it worse. At 16K tokens, the conversation is already about the size of the model. At 128K, one conversation is about 15 GB, more than seven times the model. Long chats and long-running agents are where small models do more and more of the work, and where the dial has the least to give.
Every token has a floor
So if the energy isn’t going into math, what would the math alone cost? We measured it.
We built the arithmetic an FP8 accelerator uses: a 16-wide dot product of 8-bit floating-point numbers, accumulating into FP16. We measured it the way we measure our own circuits, placed and routed, then simulated gate by gate on a real model’s data, glitches included. Each multiply-accumulate in it costs 246 femtojoules at the 5 nm class, its share of the accumulating and rounding included.
Qwen3 1.7B needs 3.6 billion multiply-accumulates to write one token at a 16K-token conversation: 1.7 billion for its weights and 1.9 billion for attention over the conversation. At 246 fJ each, that’s 0.89 mJ. That’s the floor: one token’s math, with nothing moved and nothing waiting.
Figure 3. One token’s energy against its arithmetic
Qwen3 1.7B at a 16K-token conversation, decoding only: the energy for each token on one NVIDIA B300 as one to sixteen people share it, against the FP8 arithmetic alone. The energy scale is logarithmic: each gridline is ten times the last.
Measured on an NVIDIA B300 Measured on our flow: the floor, a standard FP8 part
The B300 spends 994 mJ on that token for one person and 347 mJ for sixteen, 1,100 and 390 times the floor. Each person added saves less than the last, and by sixteen the curve has flattened out: batching doesn’t bring the GPU anywhere near the floor. The arithmetic is less than a tenth of a percent of what the GPU spends on one person’s token, and about a quarter of a percent with sixteen. And batching didn’t change the arithmetic at all. The same model writes the same token.
Where the rest goes
Energy follows time. With one person, the board draws 461 W for 2.2 ms, and most of those milliseconds are the fixed cost of stepping a chip built for a crowd through one token. With sixteen, it draws 869 W for 6.4 ms a step, and most of the step is reading conversations. Neither is math.
So the floor holds in time too. At the 461 W the board draws for one person, a token’s math alone would take 1.9 µs. It takes 2.2 ms. With sixteen people at 869 W, each person’s math would be done every 16 µs, and each waits 6.4 ms. That’s the same 1,100 and 390 times. The people waiting pay the same gap in time that the operator pays in energy.
And each new process node widens the gap. The arithmetic shrinks with the transistor, but memory and wires improve far more slowly.
No setting of the dial closes it. The fix is a chip built for the next token, not the next batch.
That means a chip sized for one person’s next token, with the model and every conversation it serves on the chip, right beside the arithmetic. Part 4 shows the design we built that way, and how we found it. First, part 3: how to read any chip’s numbers, ours included.
How we measured
Measured on an NVIDIA B300Measured on our flowCalculated
- The sweep
- One NVIDIA B300, vLLM as shipped, Qwen3 1.7B in its published FP8 checkpoint, 15K tokens in and 1K out, one to sixteen people at once, decoding only, measured on 16 October 2026. Energy is the board’s mean power over decoding time, divided by tokens. The time between a person’s tokens is the median, as in part 1; total output is the decoding rate summed over everyone. FP8 because it was the GPU’s least-energy setting; in bf16, one person’s token costs 1,176 mJ. The board drew 461 W serving one person and 869 W serving sixteen.
- vLLM as shipped
- It is what most operators run. A hand-tuned stack would be faster for one person, and every ratio against the B300 in this series would narrow there, by about half on our estimate. It would not close.
- The reads, calculated, not measured
- From Qwen3 1.7B’s published configuration: about 112 KiB of keys and values for each token of conversation, so about 1.8 GB at 16K, and about 2 GB of model read for every step. The time the reads would take is calculated at NVIDIA’s published memory bandwidth for the B300, 8 TB/s.
- The floor
- A standard FP8 dot product, the arithmetic any FP8 accelerator uses, built and measured on the same flow as every chip figure in this series: sixteen 8-bit floating-point products a cycle, accumulated into FP16, placed and routed, then simulated gate by gate on a real layer’s data, glitches included, at the 5 nm class. 246 fJ is one multiply-accumulate’s share of a 16-wide cycle, the accumulating and rounding included. The multiply-accumulates per token are from Qwen3 1.7B’s published configuration.
- The floor in time, calculated
- The time the floor’s energy would take at the board’s measured power, for everyone in the step: 0.89 mJ at 461 W for one person, sixteen times that at 869 W for sixteen. Energy is power times time, so at the same power a token’s time sits as far above the floor as its energy.
Anyone with a B300 can rerun the sweep: vLLM as shipped, with the setup above.