AI Agent Memory: Short-Term vs Long-Term Memory

State management is the difference between a simple script and a production-grade AI agent. An agent without memory is just a stateless function; an agent with memory can execute long-running workflows, pause for human input, and improve over time.

For AI Engineers building agentic systems, memory is typically divided into two distinct architectural domains: Short-Term Memory (STM) and Long-Term Memory (LTM). This reference guide breaks down the technical differences, architectures, and implementation strategies for both.

1. Primary Value Proposition

By the end of this guide, you will understand how to design agent memory systems that prevent context window exhaustion, eliminate hallucinations caused by state loss, and allow agents to resume workflows across sessions.

2. Short-Term Memory (STM)

Short-term memory is the agent’s immediate execution context. It represents the state of the current task or conversation.

The Architecture

STM is typically bounded by the LLM’s context window. It consists of the system prompt, tool schemas, and the immediate history of the current interaction loop (Observations, Thoughts, Actions).

Because context windows (even large ones like 200K tokens) are finite and expensive, STM cannot grow infinitely.

Implementation Patterns

1. Sliding Window: Only keeping the last $N$ turns of the conversation.
2. State Machine (Graph): Using frameworks like LangGraph to maintain a structured state object (e.g., Dict[str, Any]) that is updated at each node in the execution graph, rather than relying purely on raw conversational history.
3. Summarization Chains: When the context approaches a token limit, a secondary LLM call summarizes the older history into a compressed format, appending it to the top of the new context window.

Common Mistakes

• Assuming the LLM remembers everything: Context degradation occurs even within the context window limits. Information at the beginning and end is recalled better than the middle (the “Lost in the Middle” phenomenon).
• Leaking sensitive data: STM is often sent to external LLM providers. Without local redaction (like PriviPaste), PII can be exposed during the reasoning loop.

3. Long-Term Memory (LTM)

Long-term memory is persistent state that survives across sessions, agent restarts, and discrete tasks. It allows an agent to “remember” user preferences, historical decisions, and facts discovered in previous workflows.

The Architecture

LTM is fundamentally an external storage problem, typically relying on a combination of databases:

graph TD
    Agent["AI Agent"] -->|Queries| Router["Memory Router"]
    Router -->|Semantic Search| VectorDB[("Vector Database (Chroma / Pinecone)")]
    Router -->|Entity Lookup| GraphDB[("Knowledge Graph (Neo4j)")]
    Router -->|Transactional State| SQL[("Relational DB (Postgres)")]

Implementation Patterns

1. Vector Stores (Episodic Memory): Storing past conversations or document chunks as dense vectors. Useful for semantic retrieval (“What did we discuss about the deployment last week?”).
2. Knowledge Graphs (Semantic Memory): Storing entities and relationships (e.g., “User -> works at -> StackMindset”). Crucial for multi-step reasoning where exact relationships matter more than semantic similarity.
3. SQL/NoSQL (Procedural/State Memory): Storing structured data like user preferences, authentication tokens, or workflow status.

4. Bridging STM and LTM

The core challenge in agent engineering is dynamically retrieving the right LTM and injecting it into the STM at the right time.

The RAG/Agent Intersection

When an agent begins a task, it must query its LTM.

1. Tool-based Retrieval: The agent is given a tool (e.g., query_memory) and decides when to search its history.
2. Automated Injection (Context Engineering): The orchestration layer automatically queries the Vector DB based on the user’s prompt and injects the retrieved context before the agent even sees the request.

Example: Injecting Memory (Python)

def build_agent_context(user_id: str, current_prompt: str) -> list:
    # 1. Fetch structured LTM
    user_prefs = postgres.get_user_preferences(user_id)
    
    # 2. Fetch semantic LTM
    past_interactions = vector_db.search(query=current_prompt, filter={"user": user_id})
    
    # 3. Construct STM
    system_prompt = f"""
    You are an AI assistant.
    User Preferences: {user_prefs}
    Relevant Past Context: {past_interactions}
    """
    
    return [{"role": "system", "content": system_prompt}, {"role": "user", "content": current_prompt}]

5. Security & Cost Considerations

• Storage Costs: Vector databases can become expensive at scale. Implement TTL (Time To Live) on ephemeral memories.
• Context Costs: Injecting too much LTM into the STM increases the input token cost for every single reasoning step. Use rerankers to ensure only the top 3-5 most relevant memories are injected.
• Multi-Tenant Isolation: Never allow an agent to access a global memory pool without strict row-level security (RLS) tied to the current user’s execution context.

6. Key Takeaways

1. STM is for reasoning; LTM is for persistence. Do not confuse the two.
2. Structure your STM: Use explicit state objects (like LangGraph state) rather than just appending strings to a chat log.
3. Retrieve LTM dynamically: Give your agent tools to query its own past, or build middleware that injects it automatically.