A RAG that holds up in production

You know the RAG principle: split documents, index them as embeddings, retrieve the relevant passages and feed them to the model. The prototype comes together in a day. Then real users show up, and the accuracy rate disappoints. The gap between demo and production comes down to four workstreams.

1. Chunking is a decision, not a detail

Blindly splitting every 1,000 characters cuts tables in half and separates headings from their content. Serious systems split by structure: per section, with overlap between chunks, and with each chunk carrying its context (document title, section, date). A chunk must stand alone and still make sense.

2. Hybrid search

Embeddings grasp meaning ("holidays" finds "vacation") but sometimes miss the exact match: a product reference, a legal article number, a proper noun. The fix: combine vector search and classic keyword search, then merge the results. That's the production standard.

3. Reranking

Fast search brings back 30 to 50 plausible candidates. A reranker, a slower but finer-grained model, re-reads each candidate against the question and keeps only the top 5. This single stage often improves quality more than any model swap.

4. Measure retrieval separately from generation

If the final answer is poor, who's at fault: retrieval that didn't surface the right passage, or the model that misused it? Without separate measurement, you can't tell. Build a set of questions where you know the expected source passage, and check first: "did the right passage come up?" Then you fix the right stage.

Production habits

• Mandatory citations: every claim points to its source, so the user can verify.
• Allow "I don't know" when nothing relevant is found, rather than letting the model improvise.
• Live index: documents change, the index must update, or the system answers from the old version.