The short answer
Retrieval-Augmented Generation gives a language model access to your own data at question time. Rather than relying on what the model memorised during training, the system searches your documents for relevant passages, puts them into the prompt, and asks the model to answer using only that context. It is the difference between a model that guesses and a model that cites.
How a RAG pipeline actually works
A working RAG system is four stages, and each one is a place where quality is won or lost.
Ingest and chunk: Documents are split into passages small enough to be precise but large enough to keep meaning. Chunking badly is the most common cause of bad answers.
Embed and index: Each chunk becomes a vector and is stored in a vector index so semantically similar text can be found, not just keyword matches.
Retrieve: The user question is embedded and matched against the index. The top passages are selected, often re-ranked for relevance.
Generate: The retrieved passages go into the prompt with an instruction to answer only from that context, and to say so when the context does not contain the answer.
RAG or fine-tuning?
These solve different problems and are frequently confused in vendor pitches.
RAG changes what the model knows. Use it when the answer lives in documents that change — policies, product docs, tickets, contracts.
Fine-tuning changes how the model behaves. Use it when you need a consistent format, tone, or a narrow classification task.
If your content updates weekly, fine-tuning is the wrong tool: you would retrain for every change. RAG picks up new documents the moment they are indexed.
Most business problems described as "we need a custom model" are actually retrieval problems.
Where RAG quietly fails
A RAG demo works on ten documents. Production is where the failure modes appear.
Retrieval returns plausible but wrong passages, and the model confidently answers from them.
Chunks split a table or a clause in half, so the retrieved context is missing the part that mattered.
The corpus contains contradictions — an old policy and a new one — and nothing tells the model which wins.
No evaluation set exists, so nobody notices quality degrading as documents are added.
The engineering work in RAG is mostly in retrieval quality and evaluation, not in calling the model.
