AI Agent Memory Systems

The entire memory system assumes you know that "this session" belongs to the same person as "that earlier session." In practice: anonymous/logged-out sessions, users on multiple devices, shared accounts, and B2B scenarios with multiple people using one account all break this assumption. There's no general solved answer — this is treated as an open problem at the memory layer itself, meaning most systems punt it upstream to whatever authentication layer sits above them, and simply trust whatever user_id they're given.

Module 5's routing pattern (ADD/UPDATE/DELETE/NOOP) assumes a clean binary: new fact either replaces old fact, or it doesn't. Real conversations are messier — a user might state something, later say something that's ambiguous about whether it's a correction or an addition, or two different facts might both be "sort of true" depending on context (e.g., "budget is $10k" for one project, "$50k" for another — these aren't actually contradictory, they're both true but scoped differently, and a naive router might incorrectly treat the second as an UPDATE that overwrites the first). Building a good contradiction-resolution policy requires you to actually define, for your specific domain, what scoping/context a fact needs to be considered "the same fact" in the first place — this is genuinely domain-specific work, not something a generic memory library can fully solve for you.

When multiple distinct agents (say, a data-collecting agent, a review agent, a customer-facing agent) need to operate off one shared memory substrate, new problems appear that don't exist in single-agent memory: which agent's writes are authoritative if two agents write conflicting facts near-simultaneously, whether all agents should see all memories or only a filtered subset relevant to their role, and how you audit which agent wrote (or acted on) which memory when something goes wrong. This is squarely where your agent use case-specific product work lives — a generic memory library gives you the storage/retrieval primitives, but the multi-agent authority and access-control policy is something you design for your specific use case's needs.

Instead of running the expensive LLM routing call (Module 5) on every single extracted fact, first do a cheap similarity check: if the new fact's embedding is very far (below some similarity threshold) from every existing memory, it's almost certainly a genuine ADD, and you can skip the LLM call entirely and just add it directly. Only run the expensive routing LLM call when the cheap similarity check finds something close enough to be ambiguous. This can cut routing costs substantially at scale, since most new facts in a long conversation genuinely are new, not corrections.

def should_use_cheap_add(new_fact_embedding, existing_embeddings, threshold=0.3):
    if not existing_embeddings:
        return True
    max_similarity = max(cosine_similarity(new_fact_embedding, e) for e in existing_embeddings)
    return max_similarity < threshold  # far from everything = safe to just ADD directly

If the same or very similar query gets asked repeatedly (common in production — many users ask semantically similar questions), cache the retrieval result keyed by embedding similarity rather than exact string match, so a slightly-reworded repeat query can still hit the cache and skip a full retrieval + generation cycle.