Nvidia has announced full production of its inference-focused accelerator, stemming from its acquisition of Groq for around $20 billion. The chip integrates into the Vera Rubin platform, with up to 256 accelerators per rack. This is not a chip designed to train models: it is a chip designed to make them respond quickly.
We explained in our article on the two costs of AI that inference, the usage phase, now exceeds training in budgets. Here is the hardware translation of that shift.
Why a different chip
Training and serving a model do not require the same qualities, and this difference structures the entire industry.
Training demands enormous raw compute power and tolerates slowness perfectly. Whether the result arrives in six weeks or six weeks and two days makes no difference. You can therefore stack calculations in parallel without worrying about the latency of each one.
Inference demands the opposite: producing each token quickly, one after another, because a model generates word after word and each depends on the previous one. This process is sequential, hence hardly parallelisable, and it mainly hits the speed at which data can be moved between memory and the processor.
A chip optimised for training spends part of its time waiting on memory when serving a model. A chip designed for inference organises that flow differently.
Why now
Three developments make this specialisation profitable, and we have documented all of them.
Volume has exploded. With unlimited free offerings and hundreds of millions of daily users, the number of tokens served far exceeds the number of training tokens.
Agents consume a huge amount. An agent chaining steps in a loop produces and re-reads tokens constantly, as we described regarding their move into production. Platform data shows these workloads are significantly more demanding than classic conversation.
Latency has become a product criterion. We saw this yesterday with a voice model announced at under 300 milliseconds. That threshold is not reached through software alone: you need hardware capable of producing tokens fast enough.
A shift in language is worth noting. We no longer talk about raw performance but about tokens per second, tokens per watt and cost per token. These are factory indicators rather than laboratory ones: throughput, consumption, unit cost. It is the vocabulary of an industry producing continuously, and it confirms what we wrote two weeks ago about AI having become an industry like any other.
What this changes for you
Indirectly but really, two things.
Prices keep falling. More efficient inference hardware reduces the cost per request, and that drop eventually gets passed on, as we saw with this week's wave of price cuts.
Real-time uses become viable. Many interesting applications were impossible not for lack of model intelligence but because the response arrived too late: simultaneous translation, driving assistance, interaction with a robot.
The caveat
A specialised chip is excellent at its task and rigid by nature. If model architectures evolve significantly, hardware optimised for current sequential generation could age faster than general-purpose hardware.
It is a reasonable bet in the short term, token-by-token generation being the common mechanism across all current language models. It is a bet nonetheless.
What to remember
This announcement is the kind of news you skip over and which matters more than the release of yet another model.
It signals that the industry has stopped treating inference as a by-product of training and now handles it as a distinct discipline, with its own dedicated hardware. This is what happens to any technology deploying at scale: at some point, you stop adapting general tools and build the exact tool.
For an observer, it is probably the best maturity indicator available. You do not design a twenty-billion-dollar chip for a use case you doubt will last.