RAG has exactly four steps:
- Embed the user's query using the same embedding model you used to embed your documents
- Vector search — find the top-k most similar chunks in your vector store
- Construct an augmented prompt — take the retrieved chunks and insert them into the prompt, along with instructions telling the model to answer using only that context
- Generate — send the augmented prompt to the LLM, get an answer grounded in the retrieved text
import chromadb
from openai import OpenAI
client = OpenAI()
chroma_client = chromadb.Client()
collection = chroma_client.create_collection(name="rag_demo")
# Step 0: index some documents (one-time setup)
documents = [
"Our refund policy allows returns within 30 days of purchase with a receipt.",
"Shipping takes 5-7 business days for standard delivery within the US.",
"Premium support is available for Enterprise plan customers only.",
]
collection.add(documents=documents, ids=[f"doc{i}" for i in range(len(documents))])
def rag_answer(query):
# Step 1 & 2: embed + search (Chroma handles the embedding call internally here)
results = collection.query(query_texts=[query], n_results=2)
retrieved_chunks = results['documents'][0]
# Step 3: construct augmented prompt
context = "\n".join(retrieved_chunks)
prompt = f"""Answer the question using ONLY the context below.
If the answer isn't in the context, say "I don't have that information."
Context:
{context}
Question: {query}
Answer:"""
# Step 4: generate
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": prompt}]
)
return response.choices[0].message.content
print(rag_answer("How long do I have to return something?"))
print(rag_answer("What's your policy on pet adoptions?")) # should say "I don't have that"
This is the single most important conceptual distinction in this whole course, so read this twice:
- RAG's knowledge base (the
documentslist above) is static — it was written once, by you, ahead of time. The conversation doesn't change it. - Memory is built from the conversation itself, in real time. If the user says "actually my budget is $15k, not $10k," a memory system needs to write a new fact and potentially overwrite an old one — RAG has no concept of this at all, it only reads.
Every memory system you'll study from Module 4 onward is essentially "RAG, plus a write path, plus logic for handling contradictions and staleness." RAG is the read half of memory; it's necessary but not sufficient.
n_results=2 in the code above is your top-k parameter. Too low and you might miss the right chunk; too high and you dilute the prompt with irrelevant text (which both costs more tokens and can measurably confuse the model). A common production pattern: retrieve a larger set (e.g., top 20) with cheap vector search, then re-rank that smaller set with a more expensive, more accurate model (a cross-encoder, or even an LLM call) to pick the true top 3-5 before generating the answer.
- Build the exact pipeline above, run it against 10 questions, 5 answerable and 5 not — verify it correctly says "I don't have that information" for the unanswerable ones (this is the difference between a grounded system and a hallucinating one).
- Add a re-ranking step: retrieve top-10, then use an LLM call to pick which 2 are actually most relevant before generating the final answer. Compare answer quality to no re-ranking.