When context is no longer enough: seeking out the information

The Beginner level explains what the context window is and why the middle of a long text is handled less well. That leaves the practical question: what do you do when the documentation runs to ten thousand pages?

Pasting everything doesn't scale

Even with a very large window, sending ten thousand pages with every question is slow, expensive, and poor quality: the useful information is drowned in noise the model has to wade through.

The principle: search first, answer second

You split the documentation into chunks. For each question, a system first searches for the five or ten most relevant chunks, and only those are sent to the model along with the question.

This is what's called retrieval-augmented generation, or RAG. The model still doesn't know your documentation: you put it in front of it, at the right moment, in small amounts.

How the search finds things

Each chunk is turned into a list of numbers that represents its meaning, not its words. A question about "cancelling my contract" retrieves a passage about "termination of the agreement", without a single word matching. That's what sets this search apart from a simple "Ctrl+F".

Where it really matters

Contrary to what you might think, quality barely depends on the model. It depends on the chunking. Chunks that are too small lose context: you retrieve a sentence without knowing which article it comes from. Too large, and they bring back noise. Chunking that follows the document's structure, with the section title kept in each chunk, beats any model change.

The essential safeguard

Always demand the source passages along with the answer. Without them, you can't tell a grounded answer from one invented from poorly chosen chunks. An answer without a source, in this setup, is worthless.

When this isn't the solution

If the question concerns a whole corpus, "what are the main trends across these three hundred reports", chunk-based search fails by design: it returns ten passages, not an overview. In that case, you need to summarise document by document, then work on the summaries.