A large language model doesn't "run" the way a normal program does, with variables that persist in memory (RAM). Every single time you call the API, you're starting a brand new, blank conversation from the model's point of view. The only reason ChatGPT-style apps feel continuous is that the application re-sends you the entire conversation transcript every single time, and the model reads that transcript fresh, as if for the first time.
This has a very concrete consequence: if you don't explicitly include something in the prompt, the model has zero knowledge of it, no matter how many times you discussed it in "previous" turns.
Let's actually measure this instead of just asserting it.
from openai import OpenAI
client = OpenAI()
conversation = []
def chat(user_message):
conversation.append({"role": "user", "content": user_message})
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=conversation
)
reply = response.choices[0].message.content
conversation.append({"role": "assistant", "content": reply})
# print token usage to watch it grow
print(f"Tokens this turn: {response.usage.total_tokens}")
return reply
for i in range(20):
chat(f"This is turn {i}. Just say 'ok {i}'.")
Run this and watch the printed token count. It grows roughly linearly — turn 1 might cost 20 tokens, turn 20 might cost 400+ tokens, because you're re-sending everything each time. Extrapolate this to a real product with long user sessions: costs compound turn over turn, and eventually you hit the model's hard context limit (e.g., 128k or 200k tokens) and the app simply breaks or starts silently dropping old messages.
- Cost: you pay for the same old tokens again and again, every single turn.
- Context limit: eventually you run out of room entirely.
- Retrieval quality drops: even before you hit the hard limit, stuffing 50 turns of chat into the prompt means the genuinely relevant fact ("user's budget is $10k") is buried in noise, and models are measurably worse at using information buried in the middle of a long context than information near the start or end (a well-documented phenomenon sometimes called "lost in the middle").
A chatbot forgetting your name is mildly annoying. An agent forgetting:
- that it already sent an email to this lead → sends a duplicate, looks unprofessional
- that the user already tried a troubleshooting step → tells them to try it again, erodes trust
- what a customer's stated budget was three calls ago → makes a proposal that's obviously wrong
...actively breaks the product's usefulness. This is why memory is treated as load-bearing infrastructure, not a nice-to-have feature, for any serious agent product.
- Run the script above, log the token count per turn to a CSV, plot it — see the linear growth yourself.
- Now modify the script to only keep the last 3 turns in
conversation. Ask a question that requires turn-1 information (e.g., "what was the very first thing I said?") at turn 15 — watch it fail, because that information is now truly gone from what the model sees.