The work around the frontier model

Almost every new AI chip is built for the frontier model: the biggest model there is, spread across as much memory as it takes, with users batched together to keep it busy. Hardly anyone is building for the small models around it, the ones that screen your prompt, draft its answer and run an agent’s steps.

On a GPU, those models have an odd problem. Make one sixteen times smaller, and an NVIDIA B300 serving one person runs it only about three times as fast.

Small models sit around every answer

Follow one request through a modern AI service.

Before the frontier model sees your prompt, a smaller model usually screens it. Is it safe to answer? Another decides where it goes, to which model and with which tools. If the answer needs facts, small models turn your question into a search, fetch what matches and rank it.

While the frontier model writes, a small model can draft ahead of it. It guesses the next few tokens, and the big model checks a block of them in one step instead of writing them one at a time. When the answer is done, it’s screened again on the way out.

Each of those steps runs on a small model, and each sits in your critical path. Until it finishes, you wait.

Agents multiply it

An agent runs that whole path dozens of times a task. It plans, calls a tool, reads the result, decides what to do next and checks its work. Each step waits on the last.

On an NVIDIA B300, Llama 3.2 3B writes 439 tokens a second for one person. An agent that takes thirty steps, each a 300-token tool call, spends about twenty seconds waiting on that model alone. Serving more agents at once doesn’t shorten it.

Why a smaller model barely helps on a GPU

A data-center GPU is built for a big model and a crowd. Everything about it, from the memory stacks beside its die to the thousands of cores on it, pays off when one pass through the model serves many people at once. The number that matters is tokens a second, summed over everyone.

A small model serving one person is the opposite case, and we measured what it costs. On an NVIDIA B300, serving one person, Qwen2.5 0.5B writes 793 tokens a second and Llama 3.1 8B writes 260. The first model is sixteen times smaller and only about three times as fast. Across seven models from 0.5B to 8B, the pattern holds: speed barely follows size.

Fit a line through the two ends and you can see why. Every token costs the GPU about a millisecond before the model’s size even matters. That’s the time it takes to step through a long chain of small operations, one after another, with a crowd-sized chip powered up the whole time. Reading the model comes on top. For a big model, the reading is most of the time. For a small one, it’s mostly the fixed millisecond, so shrinking the model barely moves it.

Hand-built kernels can cut much of that millisecond for one person. They don’t change what the chip is built for.

The usual fix is to batch: serve more people at once, so each step does more work. That just trades one price for another. Batch, and the people waiting pay in time. Don’t, and you pay in energy for a chip that sits mostly idle. A small model on a data-center GPU has no good setting. Part 2 turns that dial, one person to sixteen, and measures both prices.

Fit the silicon to the job, not the model

One answer is dedicated silicon: a model built into the chip, lean and fast. It drops almost everything a general chip pays for, and it’s the right answer where one model will run unchanged for years. The work around the frontier model isn’t like that: its models change every few months.

Its jobs don’t. The work is settling into a handful of kinds: drafting, screening, routing, retrieval, long conversations and agent steps. Each is a job you can state, with a bar you can measure, so you can build silicon for it and say exactly how good that silicon is. Fit the silicon to the job, and the models can keep changing.

What this silicon needs

These models run up to about five billion parameters, serve one person or a few at a time, and have a person waiting on every token. That’s the opposite end of every axis a GPU is built for. Build the other way round, and the fixed millisecond has nothing to stand on. We think the silicon comes down to five things.

• Built for one person’s next token. How soon it arrives is the number that counts. Tokens a second summed over a crowd is the GPU’s number.
• The model and the conversation stay beside the math. Keep the weights, and everything each person has said, where the arithmetic happens, so no token waits on a trip to memory.
• Beside the GPU, not instead of it. The frontier model stays on the GPUs built for it. Only tokens cross between the two.
• A new model is a reload, not a new chip. The silicon is fitted to the job, so the next model loads as weights.
• Quality first. Set the bar, then count how long each person waits and how much energy each token takes. A faster answer that’s worse isn’t a win.

What we’re building

For every answer the frontier model gives, the small models around it answer many times, and each time, you wait. That work deserves silicon of its own.

We’ve designed it. Pre-silicon, it gives each of twelve people their next token 110 times sooner than a B300 serving the same twelve, at the same quality. Part 4 shows how we know.

That’s what we’re building at Exstill: the frontier model’s co-processor, a card that runs the small models beside the GPUs. We design its math with the model, and keep the whole model beside it.

The rest of the series:

• Part 2 measures what a GPU spends on one person’s token.
• Part 3 is how to judge any chip’s numbers, ours included.
• Part 4 is how we fit the silicon and the model to the work, part 5 the models built for it, part 6 puts the chip beside the GPU, drafting for the frontier model, part 7 screens every prompt and reply on it, and part 8 reasons on it in milliseconds.