A model straight out of pre-training is strangely unusable. Ask it a question and it will carry on your sentence instead of answering it. Ask it for advice and it will produce a list of similar questions. It has absorbed a prodigious amount of knowledge and it does not know it is being spoken to. Everything that makes it usable happens afterwards.
This morning we reported on a company valued at 1.4 billion that pre-trains no model at all. Here is what it does instead.
Step 1: pre-training
This is the phase everyone knows and the only one people talk about. The model learns to predict what comes next in a text across an immense corpus, as we explained in our article on training.
It costs hundreds of millions, it demands infrastructure few possess, and it produces a raw object: plenty of knowledge, no direct use.
Step 2: mid-training
This is the least known phase and often the most decisive for specialising a model.
Pre-training is continued, but on a chosen corpus: code if the target is software development, legal texts for legal use, a particular language if it was under-represented.
The model does not change its nature, it changes its centre of gravity. What was marginal in its statistics becomes central. This is how you get a model that handles technical vocabulary naturally, or that writes correctly in a poorly served language — the problem we described in our article on ignored languages.
This step costs a fraction of pre-training, and it is what makes it possible to start from an open model and turn it into something clean.
Step 3: instruction tuning
The model is shown thousands of examples of the form "here is a request, here is a good answer". It thus learns that the text it receives is a request and not the start of a paragraph to be continued.
It is this step that turns a completion engine into an assistant. It is short, inexpensive, and its effect is spectacular.
The final phase gives the model its character. Humans rank responses, those rankings become a reward signal, and the model learns to produce what is preferred — the mechanism we detailed in our article on reinforcement learning. This is where tone, caution and response length are decided. And this is where the flaws we have documented are born: the tendency to agree with you and the systematic air of confidence are not bugs, they are winning strategies under the criterion used.
Where the value lies today
Here is the practical point, and it explains the current landscape.
Step 1 has become a raw material. Several labs release excellent base models for free, and competition between them guarantees more will follow.
Steps 2 to 4 have become the product. This is where the rare expertise, the differentiation and the intellectual property lie — and it is what is sold both by a legal publisher building its own model and by a team of a hundred people valued at 1.4 billion.
This decoupling explains something that often surprises: why modest players can produce competitive models. They are not redoing the costly work, they are doing the rare work.
What to take away
When you read that a model has been "trained", ask yourself which of these phases is meant. The cost gap between the first and the ones that follow is a factor of a hundred or a thousand.
And remember the point that explains most of the current market: a model's knowledge comes from pre-training, but everything that makes it useful to you comes from what was done to it afterwards.