8. Conversation State and Memory

Up until now, every LLM interaction we've built has been stateless. Stateless means each request is independent—the model has no memory of previous conversations. This works fine for one-off tasks like document summarization or single-question Q&A.

But building a conversational agent is a different story. You need the agent to recall what was discussed earlier, understand pronouns like "it" or "that," and maintain context throughout the dialogue. To do this, you must explicitly manage state.

(Here, "state" refers to information that a program remembers. For conversational agents, the previous conversation history is the state.)

In this chapter, we'll cover:

• Why LLMs don't "remember" conversations
• How to implement conversation memory using LangChain's message history tools
• How to manage token budgets to prevent context overflow

8.1) Why LLMs Forget

LLMs Have No Memory

LLMs have a critical characteristic: they don't remember anything from previous conversations.

When you call an LLM API, the model processes your input and generates a response. But it doesn't store that record anywhere. There's no memory inside the model that maintains state, no conversation history. Each API call is completely independent. It's like starting fresh every time.

This is by design. LLMs work like stateless functions: you provide input, they produce output, and nothing is retained. The model running on OpenAI's servers right now has no record of what you just asked.

Why LLMs Seem to Remember

But wait—when you use ChatGPT or Claude, it feels like they remember your conversation. You can say "Tell me about Paris," then follow up with "What's the population?" and the model knows you're still talking about Paris. How does that work?

Here's how: the application sends previous conversation history along with each new message.

Here's what actually happens:

The LLM doesn't "remember" you asked about Paris earlier—it only knows because the application sent the previous conversation history along with the new message. Ultimately, it's the application that manages state, not the model.

Why "State" Must Be Managed in Your Application, Not the Model

Think of an LLM as a pure function: given input, it produces output. There's no internal state that the LLM manages beyond the messages you provide. This is by design.

For conversational agents, this means state must be managed in your application.

State—the conversation history—lives in your application code, not in the model. This means you're responsible for:

• Storing the conversation history
• Sending relevant history with each new request
• Managing the size of the history (covered in section 8.3)

In section 8.2, we'll implement this using LangChain's message history tools.

What Happens If You Don't Manage State?

If you don't manage state, your agent (your application) can't maintain a coherent conversation. Here are the most common failures:

1. Can't Remember Previous Conversation

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
 
llm = ChatOpenAI(model="gpt-4o-mini")
 
# First question
response1 = llm.invoke([HumanMessage(content="My name is Alice")])
print(response1.content)  # Output: Nice to meet you, Alice!
 
# Second question (no history sent)
response2 = llm.invoke([HumanMessage(content="What's my name?")])
print(response2.content)  # Output: I don't know your name...

The model has no idea you said your name was Alice because we didn't send that information in the second call.

2. Can't Tell What Pronouns Refer To

# User asks about a topic
response1 = llm.invoke([HumanMessage(content="Tell me about Python")])
print(response1.content)  
# Output: Python is a high-level programming language known for its readability...
 
# User follows up with a pronoun
response2 = llm.invoke([HumanMessage(content="What are its main features?")])
print(response2.content)  
# Output: I'd be happy to help! Could you specify what you're asking about?

Without the previous message, the model can't tell what "its" refers to.

Real-World Impact:

Imagine building a customer support agent without state management:

User: "I'm having trouble with my order #12345"
Agent: "I'm sorry to hear that. What seems to be the problem?"
User: "The shipping address is wrong"
Agent: "I can help with that. Could you provide your order number?"
User: "I just told you..."

This experience frustrates users and loses their trust. State management isn't optional for conversational agents—it's essential for creating coherent, useful interactions.

In section 8.2, we'll implement state management using LangChain's message history tools and add conversation memory to the CLI chat.

8.2) Managing Conversation State

Now that we understand why state management is critical, let's learn how to implement it. We'll use LangChain's built-in message history tools to manage conversation history. At the end, we'll apply what we learned to add state management to the CLI chat from Chapter 3.

Understanding Message Types: HumanMessage, AIMessage, and SystemMessage

Before learning how to manage conversation state, you need to know the message types used in conversation state. LangChain uses three message types to represent conversations: HumanMessage (user input), AIMessage (model response), and SystemMessage (instructions). Each message has a role (message type) and content (the actual text).

from langchain_core.messages import HumanMessage, AIMessage, SystemMessage
 
# SystemMessage: Instructions for the model's behavior
system_msg = SystemMessage(content="You are a helpful assistant specializing in Python programming.")
 
# HumanMessage: User input
user_msg = HumanMessage(content="How do I read a file in Python?")
 
# AIMessage: Model's response
# (In practice, LangChain wraps the model's response in this object - shown here for illustration)
ai_msg = AIMessage(content="You can use the `open()` function with a context manager...")

Why separate message types?

For the model to understand conversation history effectively, it needs to know the purpose of each message and who said it. The three message types serve distinct purposes:

• SystemMessage: Instructions that define how the model should behave (e.g., "Be concise", "You're a Python tutor")
• HumanMessage: What the user said
• AIMessage: What the model previously responded

This structure allows the model to distinguish between instructions, user questions, and its own past answers—which is essential for maintaining coherent multi-turn conversations.

Building Conversation History Manually

Now that we understand the three message types, let's see how to manually build a conversation history:

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, AIMessage, SystemMessage
 
llm = ChatOpenAI(model="gpt-4o-mini")
 
# Build conversation history
# SystemMessage sets instructions (once at the start)
# Then: User input → AI response → User input (natural conversation flow)
messages = [
    SystemMessage(content="You are a concise Python tutor."),
    HumanMessage(content="What's a list comprehension?"),
    AIMessage(content="A list comprehension is a concise way to create lists: [x*2 for x in range(5)]"),
    HumanMessage(content="Can you show me a more complex example?")
]
 
# Send entire history with new question
response = llm.invoke(messages)
print(response.content)

Output:

Sure! Here's a list comprehension that filters and transforms:
[x**2 for x in range(10) if x % 2 == 0]
This produces:
[0, 4, 16, 36, 64]

The model understands "a more complex example" refers to a more complex list comprehension example because we sent the full conversation history.

Using InMemoryChatMessageHistory

Building message lists manually becomes cumbersome as conversations grow. LangChain's InMemoryChatMessageHistory simplifies this by providing methods to add messages and retrieve the complete history.

Key methods:

• add_message(message): Adds a single message (HumanMessage, AIMessage, SystemMessage)
• add_messages(messages): Adds multiple messages at once
• messages: Property that returns the full list of messages
• clear(): Removes all messages (useful for starting fresh)

Example:

from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.messages import HumanMessage, AIMessage, SystemMessage
 
# Create a message history store
history = InMemoryChatMessageHistory()
 
# Add multiple messages at once
history.add_messages([
    SystemMessage(content="You are a helpful Python tutor."),
    HumanMessage(content="What's a decorator in Python?")
])
 
# Add messages one by one
history.add_message(AIMessage(content="A decorator is a function that modifies another function's behavior..."))
history.add_message(HumanMessage(content="Can you show an example?"))
 
# Retrieve all messages
messages = history.messages

Practical Example: State Management with InMemoryChatMessageHistory:

from langchain_openai import ChatOpenAI
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.messages import SystemMessage, HumanMessage
 
llm = ChatOpenAI(model="gpt-4o-mini")
history = InMemoryChatMessageHistory()
 
# Set system message and add to history (done once at the start)
system_msg = SystemMessage(content="You are a helpful Python tutor.")
history.add_message(system_msg)
 
# Chat function
def chat(user_input):
    """Process user input, send to LLM, and automatically manage conversation history"""
    # Add user input to history
    history.add_message(HumanMessage(content=user_input))
    
    # Send user input along with previous conversation history
    response = llm.invoke(history.messages)
    
    # Add model response to history
    history.add_message(response)
    
    return response.content
 
# Simulate conversation
print(chat("What's a lambda function?"))
print(chat("Show me an example"))  # Model remembers context
print(chat("What's the difference from a regular function?"))  # Still remembers

Output:

A lambda function is an anonymous function defined with the lambda keyword...
 
Here's an example: square = lambda x: x**2
You can use it like: square(5) # Returns 25
 
Lambda functions are limited to a single expression, while regular functions...

The chat() function handles state management automatically: it sends each user request along with the previous conversation history to the LLM, and adds both the request and response back to the history. Simply calling chat() maintains conversation state without additional management.

Refactoring Chapter 3: Adding Memory to Your CLI Chat

Let's take the streaming CLI chat from Chapter 3 and add conversation memory. Here's the original stateless version:

# chapter3_cli.py (original - stateless)
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
 
llm = ChatOpenAI(model="gpt-4o-mini")
 
def chat_loop():
    print("Chat started. Type 'quit' to exit.\n")
    while True:
        user_input = input("You: ")
        if user_input.lower() == "quit":
            break
        
        # Stateless - no history
        response = llm.stream([HumanMessage(content=user_input)])
        print("AI: ", end="", flush=True)
        for chunk in response:
            print(chunk.content, end="", flush=True)
        print("\n")
 
if __name__ == "__main__":
    chat_loop()

Refactored version with memory:

# chapter8_cli.py (with memory)
from langchain_openai import ChatOpenAI
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
 
llm = ChatOpenAI(model="gpt-4o-mini")
 
def chat_loop():
    print("Chat started. Type 'quit' to exit.\n")
    
    # Create history and add system message
    history = InMemoryChatMessageHistory()
    history.add_message(SystemMessage(content="You are a helpful assistant."))
    
    while True:
        user_input = input("You: ")
        if user_input.lower() == "quit":
            break
        
        # Add user message to history
        history.add_message(HumanMessage(content=user_input))
        
        # Stream response
        print("AI: ", end="", flush=True)
        full_response = ""
        for chunk in llm.stream(history.messages):
            print(chunk.content, end="", flush=True)
            full_response += chunk.content
        print("\n")
        
        # Add AI response to history
        history.add_message(AIMessage(content=full_response))
 
if __name__ == "__main__":
    chat_loop()

What changed:

1. Added history storage: Created InMemoryChatMessageHistory() instance inside chat_loop()
2. System message: Added to history once at the start
3. Send user input with history: llm.stream(history.messages) includes previous conversation
4. Track conversation: Add both user input and LLM response to history

Testing the refactored chat:

Chat started. Type 'quit' to exit.
 
You: My name is Alice
AI: Nice to meet you, Alice! How can I help you today?
 
You: What's my name?
AI: Your name is Alice.
 
You: What did I just ask you?
AI: You asked me what your name is.
 
You: quit

The model now maintains context across the entire conversation. It remembers your name, previous questions, and can reference earlier parts of the dialogue.

Persistent Storage: Moving Beyond In-Memory Options

InMemoryChatMessageHistory is convenient for local development, but moving to production requires replacing it with a storage solution that guarantees persistence.

Technical Limitations of InMemory:

• Volatile RAM: When the server process terminates or restarts, all conversation history stored in memory is immediately deleted. Updates or error recovery result in complete loss of user context.
• No horizontal scaling: As your service scales to multiple server instances, each server maintains its own isolated memory. Users connecting to different servers cannot share conversation history.
• Resource Inefficiency: Storing all conversation history in RAM is memory-intensive and threatens system stability as concurrent users increase.

Professional Alternatives:

• PostgresChatMessageHistory (Recommended): The most robust and widely adopted choice. Uses PostgreSQL for permanent storage and excels at complex queries and data analysis.
• SQLChatMessageHistory: Leverages existing SQL databases like MySQL. Allows you to use your current infrastructure without changes.
• RedisChatMessageHistory: Ideal for services where response speed is critical. Memory-based with persistence options, specialized for high-traffic handling.

"Storage changes, code stays the same"

LangChain provides a unified interface across all storage backends. The same methods you used with InMemoryChatMessageHistory—like add_message() and add_messages()—work identically with other storage options. This means your business logic (conversation handling code) requires no changes when switching storage.

# [Development] Local in-memory
# from langchain_core.chat_history import InMemoryChatMessageHistory
# history = InMemoryChatMessageHistory()
 
# [Production] PostgreSQL persistent storage
import psycopg
from langchain_postgres import PostgresChatMessageHistory
 
# Create PostgreSQL connection
sync_connection = psycopg.connect(
    "postgresql://user:password@10.1.1.100:5432/agent_db",
    autocommit=True,
)
 
# Specify DB connection and session ID
# session_id: Unique key to identify conversations
#   - Per-user management: session_id = user_id (one conversation per user)
#   - Per-session management: session_id = uuid (new ID for each conversation)
history = PostgresChatMessageHistory(
    table_name="chat_history",
    session_id="user_123",
    sync_connection=sync_connection
)
 
# --- Same interface regardless of storage type ---
history.add_message(HumanMessage(content="Show me my previous order."))
print(history.messages)

Important: This guide uses InMemory for quick progression, but production deployments must switch to persistent storage like PostgresChatMessageHistory.

8.3) Managing Conversation Length

The conversation memory feature we implemented in the previous section has an important problem: it only adds messages to the history. This means the history keeps growing, which creates two major issues:

1. Cost: The entire history is sent with every request, so cost per request continues to increase as history grows
2. Context window limits: Models have a maximum input size they can process in a single request (e.g., 400K tokens for GPT-5). When conversation history exceeds this limit, the model cannot process the request properly.

One of the simplest solutions is the sliding window pattern.

Sliding Window Pattern (Keep Only the Last N Messages)

The sliding window pattern solves the two problems above by keeping only the most recent N messages in the conversation history. It manages history size by discarding old messages, providing the following benefits:

1. Cost control: By keeping history below a certain size regardless of conversation length, it prevents per-request costs from growing infinitely
2. No overflow: Keeps input size within the model's maximum limit

Conceptual diagram:

With a window size of 4, we keep only the most recent 4 messages (3, 4, 5, 6) and discard the older messages (1, 2). When a new message (7) arrives, the window moves toward the latest message, dropping the oldest message (3) and keeping messages 4, 5, 6, 7.

Sliding Window Trade-offs:

• Pros: Limits history size to keep costs constant and prevents context window overflow
• Cons: Messages beyond the window size are discarded, so the model cannot reference them

This trade-off can be problematic. The solution is to use a sliding window for recent conversation while retrieving necessary past information from a separate storage when needed. This can be implemented using RAG (Retrieval-Augmented Generation), which we'll cover in Chapter 9.

Window Size Units: Message Count vs Token Count

The diagram above shows an example of setting window size by message count. However, in production environments, token-based window sizing is more commonly used because message sizes vary:

Message Count-Based Trimming:

Limits window size by message count (e.g., keep only the last 20 messages).

• Characteristics: Fixed message count, but total token count can still vary
• Use when: Message sizes are controlled (SMS, character-limited chats)
• Risk: One long message can still exceed context window

Token Count-Based Trimming (Recommended for Production):

Limits window size by token count, the input unit processed by LLMs (e.g., keep only the last 5,000 tokens).

• Characteristics: Never exceeds context window regardless of message length
• Use when: Message sizes vary

Implementing Token-Based Trimming: trim_messages()

LangChain provides a trim_messages() utility that implements the token-based sliding window pattern.

How trim_messages() works:

This function takes the full message list and maximum token count, returning only the most recent messages that fit within the token limit.

Key parameters:

• messages: Message list to trim
• max_tokens: Maximum token count to maintain
• token_counter: Function to calculate token count for each message (uses the model's tokenizer to return message token count)
• include_system: Whether to always keep SystemMessage (usually True)

Setting up the token counter:

import tiktoken
from langchain_core.messages import BaseMessage, HumanMessage, AIMessage, SystemMessage
from langchain_core.messages.utils import trim_messages
 
# Get tokenizer for your model (different model families use different tokenizers)
# Provide model name to get the appropriate tokenizer
# Claude models: Use Anthropic's tokenizer (tiktoken is OpenAI-specific)
enc = tiktoken.encoding_for_model("gpt-4o")
 
def token_counter(msg: BaseMessage) -> int:
    """Count tokens in a single message."""
    return len(enc.encode(msg.content or ""))
 
# Create conversation history
messages = [
    SystemMessage(content="You are a helpful assistant."),
    HumanMessage(content="Hi!"),
    AIMessage(content="Hello! How can I help?"),
    HumanMessage(content="What's 2+2?"),
    AIMessage(content="2+2 equals 4."),
    HumanMessage(content="What's 3+3?"),
    AIMessage(content="3+3 equals 6."),
    HumanMessage(content="What's 4+4?"),
]
 
# Keep only messages within max token count
trimmed = trim_messages(
    messages,
    max_tokens=30,
    token_counter=token_counter,
    include_system=True,
)
 
print(f"Original: {len(messages)} messages")
print(f"Trimmed: {len(trimmed)} messages")
for m in trimmed:
    print(f"{type(m).__name__}: {m.content}")

Note: The number of messages retained depends on the max_tokens value and actual token count of each message. In the example above, max_tokens=30 is a very small value chosen for testing purposes. In production, you should set an appropriate value considering average message size and desired conversation range.

Example: Applying Trimming to Chat Function

import tiktoken
from langchain_openai import ChatOpenAI
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.messages import SystemMessage, HumanMessage, BaseMessage
from langchain_core.messages.utils import trim_messages
 
llm = ChatOpenAI(model="gpt-4o-mini")
history = InMemoryChatMessageHistory()
enc = tiktoken.encoding_for_model("gpt-4o-mini")
 
def token_counter(msg: BaseMessage) -> int:
    """Count tokens in a single message."""
    return len(enc.encode(msg.content or ""))
 
def chat_with_trimming(user_input: str, max_tokens: int = 1000) -> str:
    """Chat with automatic history trimming."""
    history.add_message(HumanMessage(content=user_input))
    
    system_msg = SystemMessage(content="You are a helpful assistant.")
    all_messages = [system_msg] + history.messages
    
    # Trim to max token count
    trimmed_messages = trim_messages(
        all_messages,
        max_tokens=max_tokens,
        token_counter=token_counter,
        include_system=True,
    )
    
    response = llm.invoke(trimmed_messages)
    history.add_message(response)
    
    return response.content
 
# Usage example
print(chat_with_trimming("I'm planning a trip to Japan"))
print(chat_with_trimming("What should I visit in Tokyo?"))
print(chat_with_trimming("How many days should I spend there?"))
# Even as history grows, only recent conversation within max token count is sent to LLM

Next Step: Limitations of Conversation State Management and Solutions (Chapter 9: RAG)

Providing conversation history to the model helps maintain conversation context. However, this alone is not sufficient in some cases. For example:

• When you need to find information in company documents or manuals
• When you need to reference old conversation history that has been pushed out of the sliding window

This is where RAG (Retrieval-Augmented Generation) is needed. RAG works as follows:

1. Storage: Store information in a vector database for semantic search
2. Retrieval: Query information with similar meaning to what you're looking for

If using RAG to complement the sliding window pattern's limitations:

• Sliding window: Keep last 20 messages (recent context)
• RAG: Search and retrieve relevant content from messages pushed out of the window

Rather than just remembering recent conversation, RAG allows you to create a long-term memory system.

RAG enables agents to leverage broader knowledge and longer context through external knowledge utilization and past conversation retrieval.

Chapter 9 will cover how to implement RAG in detail.

Chapter Summary:

In this chapter, you learned:

1. Why LLMs forget: Models are stateless—memory is an illusion created by re-sending conversation history
2. Message types: SystemMessage (instructions), HumanMessage (user input), AIMessage (model responses)
3. State management: Managing conversation history with InMemoryChatMessageHistory
4. Conversation length management: Why unlimited history causes cost and context window problems
5. Sliding windows: A pattern that keeps only recent messages to prevent history from growing infinitely
6. Token-based trimming: Implementing sliding windows using trim_messages() and token counting