From raw text to assistant: fine-tuning and RLHF

A model just pre-trained on the internet is a raw completion machine: ask it “What is the capital of France?” and it might answer… with three more geography questions, because lists of questions abound in its data. To turn it into a useful, safe assistant, two more steps are needed.

Step 2: supervised fine-tuning

The model is re-trained on tens of thousands of exemplary dialogues, often written by humans: question → ideal answer. The model learns the assistant format: answer the question asked, structure the response, adopt the right tone.

Step 3: RLHF

RLHF (reinforcement learning from human feedback) refines behaviour: the model generates several answers to the same question; humans rank them from best to worst; a reward model learns to predict these preferences; then the LLM is optimised to maximise that reward. Result: more useful, more honest, less harmful. This is what turned lab curiosities into products everyone can use, and it is also what gives AIs their tendency to flatter, because humans rate what pleases them more highly.

Alignment, the ongoing challenge

Making the AI do what we really want, not just what we measured, is called alignment. Refusing to help build a weapon but explaining chemistry to a student; being honest without being hurtful; helping without flattering: these balances are set by these steps, and no lab considers them solved.