Episodic and Semantic Memory Systems

Many useful agents need both episodic and semantic memory. Episodic memory remembers what happened. Semantic memory retrieves what is known.

A customer support agent might need episodic memory to know that a user already contacted support yesterday. It might need semantic memory to retrieve the official refund policy. A research agent might need episodic memory to track which sources it already reviewed and semantic memory to query its growing knowledge base.

The two memory types answer different questions:

Episodic memory: What happened before?
Semantic memory: What information is relevant?

Episodic memory design

Episodic memory should be structured. Storing raw transcripts forever is rarely the best option. Instead, store concise event records.

Example support episode:

{
  "memory_type": "episodic",
  "user_id": "user_123",
  "episode_id": "ep_2026_06_04_001",
  "timestamp": "2026-06-04T15:22:00Z",
  "summary": "User asked about delayed order ORD-7711. Shipping status showed delayed. Agent explained refund eligibility and created a refund request draft.",
  "entities": {
    "order_id": "ORD-7711",
    "issue_type": "delayed_delivery"
  },
  "outcome": "refund_request_draft_created",
  "sensitivity": "customer_support"
}

This is easier to retrieve and safer than storing every message verbatim.

When to write episodic memory

An agent should not write memory after every sentence. Define write triggers.

Good triggers include:

• Task completed
• User gave durable preference
• Important decision made
• Ticket or workflow created
• User corrected the agent
• Action taken through a tool
• Long-running task state changed

Example policy:

def should_write_episode(event):
    return event.type in {
        "task_completed",
        "user_preference_confirmed",
        "tool_action_completed",
        "important_decision"
    }

Memory writes should be intentional.

Retrieving episodic memory

Episodic memory retrieval may use exact filters, semantic search, or both.

For customer support, exact filters are often important:

def get_recent_support_episodes(user_id, limit=5):
    return db.query(
        """
        SELECT * FROM support_episodes
        WHERE user_id = ?
        ORDER BY timestamp DESC
        LIMIT ?
        """,
        [user_id, limit]
    )

For a research assistant, semantic retrieval may be useful:

Find past research episodes related to vector databases and retrieval evaluation.

The best approach depends on whether you need exact continuity or meaning-based recall.

Semantic memory design

Semantic memory stores knowledge chunks. It should include text, embeddings, metadata, and source information.

Example semantic record:

{
  "memory_type": "semantic",
  "chunk_id": "refund_policy_004",
  "text": "Refund requests for late deliveries may be submitted after five business days past the promised date.",
  "metadata": {
    "document": "Refund Policy",
    "version": "2025-11",
    "department": "customer_support",
    "access_level": "support_staff"
  }
}

Metadata makes memory governable. It supports filtering by version, access level, department, source type, and freshness.

Combining episodic and semantic memory

A support agent query might use both:

User: What is happening with my delayed order?

Retrieval plan:

1. Retrieve recent episodic memories for this user.
2. Retrieve order status from tool.
3. Retrieve refund policy from semantic memory.
4. Build context from all three.
5. Answer with current status and next step.

Prompt context might include:

Recent episode:
Yesterday, user asked about ORD-7711. Package was delayed and refund request draft was discussed.

Current order status:
ORD-7711 is still delayed.

Relevant policy:
Refund request may be submitted after five business days past promised delivery date.

Now the agent can respond with continuity and policy grounding.

Memory conflicts

Memory can conflict. An old episode may say the user prefers email updates, while a recent message says they now prefer SMS. A policy document may have a newer version that supersedes an older one.

Conflict policy matters:

Prefer newer explicit user preferences over older ones.
Prefer approved policy documents over episode summaries.
Prefer current tool results over old episodic memory.
If conflict remains, state uncertainty or ask the user.

Without conflict rules, agents may mix outdated and current information.

Access control

External memory must respect permissions. A user should not retrieve another user’s episodic memory. An employee should not retrieve documents above their access level.

Retrieval should filter before results enter the prompt:

def retrieve_policy_chunks(query, user_access_level):
    return vector_db.search(
        query=query,
        filters={"access_level": {"$lte": user_access_level}},
        top_k=5
    )

Do not rely on the model to ignore unauthorized information. The application should prevent it from seeing that information.

Practical takeaway

Episodic and semantic memory solve different problems. Episodic memory gives continuity across events. Semantic memory gives access to durable knowledge. Strong agents often need both.

Design each memory type with clear write policies, retrieval policies, metadata, freshness rules, and access controls. Memory should make the agent more useful without making it less safe or less accurate.