Core

RAG (Retrieval Augmented Generation)

The full pipeline, explained step by step

0%

complete

5 sections·~3 min read·1 expanded

RAG has exactly four steps:

  1. Embed the user's query using the same embedding model you used to embed your documents
  2. Vector search — find the top-k most similar chunks in your vector store
  3. Construct an augmented prompt — take the retrieved chunks and insert them into the prompt, along with instructions telling the model to answer using only that context
  4. Generate — send the augmented prompt to the LLM, get an answer grounded in the retrieved text
import chromadb
from openai import OpenAI

client = OpenAI()
chroma_client = chromadb.Client()
collection = chroma_client.create_collection(name="rag_demo")

# Step 0: index some documents (one-time setup)
documents = [
    "Our refund policy allows returns within 30 days of purchase with a receipt.",
    "Shipping takes 5-7 business days for standard delivery within the US.",
    "Premium support is available for Enterprise plan customers only.",
]
collection.add(documents=documents, ids=[f"doc{i}" for i in range(len(documents))])

def rag_answer(query):
    # Step 1 & 2: embed + search (Chroma handles the embedding call internally here)
    results = collection.query(query_texts=[query], n_results=2)
    retrieved_chunks = results['documents'][0]

    # Step 3: construct augmented prompt
    context = "\n".join(retrieved_chunks)
    prompt = f"""Answer the question using ONLY the context below. 
If the answer isn't in the context, say "I don't have that information."

Context:
{context}

Question: {query}
Answer:"""

    # Step 4: generate
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

print(rag_answer("How long do I have to return something?"))
print(rag_answer("What's your policy on pet adoptions?"))  # should say "I don't have that"

This is the single most important conceptual distinction in this whole course, so read this twice:

  • RAG's knowledge base (the documents list above) is static — it was written once, by you, ahead of time. The conversation doesn't change it.
  • Memory is built from the conversation itself, in real time. If the user says "actually my budget is $15k, not $10k," a memory system needs to write a new fact and potentially overwrite an old one — RAG has no concept of this at all, it only reads.

Every memory system you'll study from Module 4 onward is essentially "RAG, plus a write path, plus logic for handling contradictions and staleness." RAG is the read half of memory; it's necessary but not sufficient.

n_results=2 in the code above is your top-k parameter. Too low and you might miss the right chunk; too high and you dilute the prompt with irrelevant text (which both costs more tokens and can measurably confuse the model). A common production pattern: retrieve a larger set (e.g., top 20) with cheap vector search, then re-rank that smaller set with a more expensive, more accurate model (a cross-encoder, or even an LLM call) to pick the true top 3-5 before generating the answer.

  1. Build the exact pipeline above, run it against 10 questions, 5 answerable and 5 not — verify it correctly says "I don't have that information" for the unanswerable ones (this is the difference between a grounded system and a hallucinating one).
  2. Add a re-ranking step: retrieve top-10, then use an LLM call to pick which 2 are actually most relevant before generating the final answer. Compare answer quality to no re-ranking.