Spending compute at inference time

For years, improving a model meant training it more: more data, more parameters, more compute upfront. That path still yields gains, but they are expensive and slow to obtain.

The shift

The other lever is to spend compute at response time. The same model, given the right to produce ten thousand tokens of intermediate reasoning before concluding, solves problems it misses when answering directly.

Nothing has been retrained. Only the amount of compute spent per question has changed. This is what is called inference-time compute.

Why it works

A model produces one token at a time, and each token represents a fixed amount of compute. Answering "47" immediately to a four-step problem requires fitting everything into a single pass. By writing out the steps, the model grants itself additional passes and uses its own text as working memory.

The written reasoning is not an explanation for you: it is the model's compute space. That is why it is often hidden, it is not meant to be read.

What it changed in practice

The clearest gains are in mathematics, code, and multi-constraint problems: where a correct answer exists and a single failed step invalidates everything. For rewriting, summarising, or conversation, the gain is close to zero.

The economic consequence

A model that produces ten thousand invisible tokens before answering bills for them. One question can cost a hundred times an ordinary question. Hence the trade-off embraced by all providers: two model families, and you choose.