The problem RAG solves
A base LLM has three blind spots for business use:
- It doesn’t know your private data — contracts, tickets, wikis and databases were never in its training set.
- Its knowledge is frozen at a training cutoff date.
- It can’t show its sources, so you can’t verify an answer — and it may hallucinate one.
RAG fixes all three by turning the task from a closed-book exam into an open-book exam: find the right pages first, then answer from them.
How RAG works: two pipelines
Pipeline 1 — Indexing (offline, runs when documents change)
Documents are parsed, split into chunks (see chunking strategies), turned into embeddings and stored with metadata such as source, section, date and access permissions.
Pipeline 2 — Retrieval and generation (online, every question)
Interactive: follow one question through RAG
Minimal RAG in Python
A framework-free sketch so you can see every moving part. In production you’d use a vector database and a proper document parser, but the logic is identical.
# pip install numpy (plus your provider's SDK for embed() and generate())
import numpy as np
docs = load_and_chunk("policies/") # -> list of {"text": ..., "source": ...}
vecs = np.array([embed(d["text"]) for d in docs]) # index once, offline
def retrieve(question, k=4):
q = np.array(embed(question))
sims = vecs @ q / (np.linalg.norm(vecs, axis=1) * np.linalg.norm(q)) # cosine
top = sims.argsort()[::-1][:k]
return [docs[i] for i in top if sims[i] > 0.5] # score threshold
def answer(question):
ctx = "\n\n".join(f"[{d['source']}]\n{d['text']}" for d in retrieve(question))
prompt = f"""Answer using ONLY the context. Cite sources in [brackets].
If the answer is not in the context, say "I don't know based on the documents."
<context>
{ctx}
</context>
Question: {question}"""
return generate(prompt, temperature=0)Want to see how chunk size changes what gets retrieved? Paste a document into the RAG Chunking Visualizer.
Advanced RAG techniques
| Technique | What it does | When it helps |
|---|---|---|
| Hybrid search | Combines vector similarity with keyword (BM25) matching | Exact terms: product codes, error IDs, names |
| Re-ranking | A cross-encoder re-scores the top ~20–50 candidates | Right chunk is retrieved but ranked too low |
| Metadata filtering | Restrict search by date, region, product, permissions | Multiple versions/regions of similar docs |
| Query rewriting | LLM rewrites vague or conversational queries before search | Follow-ups like “what about for contractors?” |
| Parent-child chunks | Search small chunks, return their larger parent section | Precise matching but answers need surrounding context |
| Agentic RAG | An agent decides when and where to retrieve, and retries | Multi-hop questions across several sources |
| GraphRAG | Builds a knowledge graph of entities and relationships | “Big picture” questions across many documents |
How to evaluate a RAG system
Build a set of 50–200 real questions with known answers and source passages. Then score the two halves separately:
- Retrieval: recall@k (is the right chunk in the top k?), precision and MRR (how high is it ranked?).
- Generation: faithfulness (is every claim supported by the retrieved context?), answer relevance and correctness versus the reference answer.
Open-source frameworks such as Ragas and DeepEval automate many of these metrics using LLM-as-judge scoring; spot-check the judge against human ratings before trusting it.
Why RAG fails (and how to fix it)
| Symptom | Likely cause | Fix |
|---|---|---|
| “I don’t know” when the answer exists | Retrieval missed the chunk | Hybrid search, better chunking, query rewriting, higher k + re-ranking |
| Answer mixes up versions | Old and new docs both retrieved | Metadata filters, de-duplicate, archive stale docs |
| Half-correct answers | Chunk cut mid-explanation | Structure-aware chunks, overlap, parent-child retrieval |
| Confident wrong answer | No abstain rule, irrelevant chunks | Score threshold, “answer only from context” + “say I don’t know” |
| Tables and scanned PDFs garbled | Weak parsing | Layout-aware parsers or OCR; store tables as Markdown |
Real-world RAG use cases
- Internal knowledge assistants over HR, IT and policy documents.
- Customer support answers drafted from help-centre articles and past resolved tickets.
- Insurance and legal clause lookup and contract Q&A with citations.
- Analytics “chat with your data dictionary” that grounds text-to-SQL in real table definitions.
- Sales enablement assistants over product sheets and case studies.
Ready to build one end to end? The RAG Engineer path walks you through each stage with hands-on labs, and AI on the Job has realistic tickets such as claims-document Q&A.
Discussion