How to read an AI chip number

Every AI chip leads with a number. Most of those numbers arrive before the chip does, and many measure something that’s easy to move. Here are three questions to ask of any of them, ours included.

What does it reward?

Tokens per watt, tokens a joule by another name, is the number most AI chips lead with. It replaced FLOPS per watt, which was worse. FLOPS count the arithmetic a chip could do, not the arithmetic it does, and when one person is waiting most of it sits idle (part 2). Tokens per watt at least counts output. But look at what moves it:

• Batch more people. The number goes up. Everyone waits longer.
• Slow the chip down. Each token costs less energy. It also arrives later.
• Shrink or compress the model too far. Tokens get cheaper. They also get worse.
• Use a tokenizer that chops words finer. The same sentence becomes more tokens. It looks like more work. It isn’t.

Most of these are good engineering. They just aren’t wins for the person waiting.

Chip designers hit the same problem in the 1990s. Ricardo Gonzalez and Mark Horowitz pointed out that if you count only energy, going slow wins, and if you count only speed, burning power wins. So they multiplied the two: energy × delay. A change that saves energy by making people wait, or saves time by burning power, has to pay for itself in the product.

Score part 2’s sweep both ways. From one person to sixteen, tokens per joule went from 1.0 to 2.9, nearly three times better by the usual measure. Energy × delay went from 2.14 joule-milliseconds a token to 2.20. Batching left it where it started. By this measure, the best setting for this GPU is four people.

Then add one rule: set the quality bar first, and measure it on the text, not the tokens. We use bits per byte, the model’s total surprise at a text, in bits, divided by the text’s length in bytes. A finer tokenizer can’t flatter it. Now ask for the least energy × delay that clears the bar, on any model and any chip. You can’t win that with a worse answer or a slower one.

How far down did the number go?

A chip takes a year or more to design, so most of its numbers are pre-silicon. That’s normal. The question is how much detail stands behind one. Before silicon, a chip can be costed at four levels:

• Paper. Math units times clock speed. It’s the ceiling, the arithmetic the chip could do if every unit were busy every cycle. For one person’s next token, a chip rarely gets near it.
• Architecture. The workload run on a software sketch of the chip, with an assumed cost for each step. Good enough to choose between ideas, not good enough to promise a number.
• Cycle. The chip’s real design, simulated one clock cycle at a time, running its actual program. The time is exact. The energy is still an estimate.
• Gate. The circuits placed and routed as they’ll be made, with the energy of every gate that switches. Down to the picojoule.

Then comes silicon, where anything the levels above missed shows up, after the money is spent.

Did the whole model run?

Below Architecture, the simulated chip has to run its actual program, and turning a whole model into that program takes a compiler. Many chips won’t have one before silicon. So what reaches the lower levels is usually one kernel, written by hand: the big matrix math the chip was built for, multiplied up as if the rest of the model ran just as well. It rarely does. The rest of the model, and the data moving between layers, is where chips lose time and energy. On a GPU serving sixteen people, part 2 found that reading their conversations costs more than reading the model.

Even a company with shipping silicon often can’t fully cost a new architecture, because the compiler and kernels for it don’t exist yet.

A number misleads when it travels without its level.

Our labels

Every figure we give says how we know it, with one of five labels:

• Measured on an NVIDIA B300. On our own bench, with the setup beside each figure, so anyone with a B300 can rerun it.
• Measured, the benchmark or block named. A model’s quality on the benchmark the label names, or one circuit block measured on our gate-level flow, placed, routed and simulated gate by gate, as part 2’s FP8 arithmetic was. Never a whole chip: that’s pre-silicon.
• Calculated. From a model’s public configuration or a published specification, not measured: what a token reads, or how long the reads would take.
• Pre-silicon 5 nm class. Our chip’s figures. Every circuit is placed, routed and simulated gate by gate, and the figures cover the whole model, every layer, not one kernel multiplied up. They’re our estimate until silicon.
• Simulated. A gain that combines a B300 measurement with a pre-silicon figure.

Part 4 shows the design behind those answers, and how it was found.