11. Conversational RAG: Adding Memory to Retrieval

In Chapter 10 we improved the retrieval quality of our RAG system. But there is still one limitation: every question is treated individually. When a user asks "What is your refund policy?", our system finds the relevant content from the documents and answers. When the next question comes in, the system answers it with no memory of the previous conversation.

Let's see why this is a problem in a real conversation. A user asks "What is your refund policy?" and then follows up with "Does that apply to digital products too?" This follow-up assumes the "refund policy" context from the previous turn, but the question text itself contains no such information. If we use "Does that apply to digital products too?" directly as a search query, the retriever will pull irrelevant information related to "digital products" (for example, pricing or specifications), and the RAG system will generate an answer that doesn't match the user's intent.

In this chapter, we'll learn how to solve this problem. We'll learn to rewrite ambiguous follow-up questions into complete questions, use that technique to build a conversational RAG system, and cover how to manage conversation history as conversations grow longer.

11.1) Rewriting Follow-up Questions into Complete Questions

As we saw in the introduction, follow-up questions build on the context of previous conversation, so people tend to omit much of the information. As a result, a follow-up question is often incomplete on its own. How can we solve this?

In Chapter 8 we learned how to help an LLM understand conversation context by passing the conversation history along with each message. We can apply the same approach here. We pass the follow-up question together with the conversation history to the LLM, and ask it to rewrite it into a complete question that reflects the context. For example, the follow-up "Does that apply to digital products too?" is rewritten, along with the conversation history, into "Are digital products eligible for a refund?" With this rewritten question, the search can find the right documents about refund policies for digital products.

This technique is called query rewriting. Let's create a system prompt for it.

from langchain_openai import ChatOpenAI
from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
 
llm = ChatOpenAI(model="gpt-5-mini")
 
system_prompt = (
    "Given a chat history and the latest user question "
    "which might reference context in the chat history, "
    "formulate a standalone question "
    "which can be understood without the chat history. "
    "Do NOT answer the question, just reformulate it if needed "
    "and otherwise return it as is."
)

The core instruction in this system prompt is "rewrite the follow-up question into a complete question using the chat history." Two specific directives are important.

First, "Do NOT answer the question, just reformulate it." This tells the LLM to only rewrite the question, not answer it. Without this directive, the LLM tends to answer the question instead of rewriting it. What we want here is not an answer, but a complete question that can be understood without the chat history.

Second, "otherwise return it as is." This tells the LLM to leave the question unchanged if it doesn't need rewriting. Without this, the LLM may unnecessarily rephrase the question, potentially changing its original meaning or scope.

Now let's use this system prompt to actually rewrite a follow-up question.

messages = [
    SystemMessage(content=system_prompt),
    # Chat history
    HumanMessage(content="What is your refund policy?"),
    AIMessage(content="All physical products may be returned within 30 days of purchase for a full refund."),
    # Follow-up question
    HumanMessage(content="Does that apply to digital products too?"),
]
 
response = llm.invoke(messages)
print(response.content)

Output:

Are digital products eligible for a refund?

The LLM read the conversation history, recognized the question was about "refund policy," and rewrote it into a complete question. Searching with this rewritten question will return documents that match the user's intent.

In the next section, we'll integrate this rewriting step into the RAG pipeline so that rewriting, retrieval, and answer generation all happen in a single call.

11.2) Building Conversational RAG

In the previous section we learned how to rewrite follow-up questions into complete questions by passing conversation history to the LLM. Now we'll integrate this rewriting step into the RAG pipeline to build a conversational RAG where rewriting → retrieval → answer generation all happen in a single call.

LangChain provides chain utilities for building conversational RAG (create_history_aware_retriever, create_retrieval_chain, etc.), but these functions are in the langchain-classic package, which reaches end of support in December 2026. The official LangChain documentation now recommends using agents instead.

We'll therefore use agents to implement conversational RAG in this chapter. Agents are covered in detail in Part V (Chapters 15–17), so here we'll only introduce what's needed for our conversational RAG implementation.

11.2.1) Agent Components We'll Use Here

In Chapter 5 we got a brief look at the core concept of agents. When the LLM analyzes a user's request and decides which tool to use, the system executes that decision. Back then we implemented this process manually, but LangChain provides APIs that make it much simpler. Here's a brief introduction to the three components we'll use.

@tool: A decorator that converts a regular Python function into a tool the agent can use. The agent autonomously selects and calls the appropriate tool from its registered tools based on the user's request.

create_agent: A function that takes an LLM, a list of tools, and a system prompt to create an agent. It handles the agent's decision-execution flow internally.

InMemorySaver: A checkpointer that automatically manages conversation history. It organizes conversations by thread_id, so when the agent is invoked with the same thread_id, it automatically loads the previous conversation history.

11.2.2) Creating the Retrieval Tool

First, let's turn the vector store search we built in Chapter 10 into a tool the agent can use.

from langchain.tools import tool
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
 
# Connect to the vector store built in Chapter 10
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = Chroma(
    persist_directory="data/chroma_db",
    collection_name="company_docs",
    embedding_function=embedding_model,
)
 
@tool
def retrieve_context(query: str):
    """Search documents for content relevant to the query."""
    retrieved_docs = vector_store.similarity_search(query, k=3)
    serialized = "\n\n".join(
        f"Source: {doc.metadata['source']}\nContent: {doc.page_content}"
        for doc in retrieved_docs
    )
    return serialized

The @tool decorator converts the retrieve_context function into a tool the agent can use. The agent autonomously decides whether to call this tool based on the user's question.

11.2.3) Creating the Agent

We pass the retrieval tool, a system prompt, and a checkpointer to create_agent to create the agent.

from langchain.agents import create_agent
from langgraph.checkpoint.memory import InMemorySaver  # Installed automatically with langchain
 
agent = create_agent(
    model="gpt-5-mini",
    tools=[retrieve_context],
    system_prompt=(
        "You are a helpful assistant that answers questions about company policies. "
        "Use the retrieve_context tool to search for relevant information. "
        "If the retrieved context does not contain relevant information, "
        "say that you don't know. "
        "Keep the answer concise, three sentences maximum."
    ),
    checkpointer=InMemorySaver(),
)

• model: The LLM the agent will use.
• tools: The list of tools available to the agent. We register the document retrieval tool (retrieve_context) we created above.
• system_prompt: The agent's behavioral instructions. It tells the agent to use the retrieval tool to answer questions about company policies, and to say it doesn't know when the retrieved context lacks relevant information.
• checkpointer: Automatically manages conversation history. InMemorySaver() stores conversations in memory, automatically handling the conversation history we managed manually in Chapter 8.

When the agent receives a user question, it consults the conversation history and decides whether a document search in the vector store is needed. If so, it calls the retrieve_context tool to fetch relevant documents and generates an answer through the LLM. The conversation history is automatically managed by InMemorySaver.

11.2.4) Running a Multi-turn Conversation

Let's run an actual two-turn conversation to verify it handles follow-up questions correctly.

# thread_id is an identifier that distinguishes conversations
# Using the same thread_id continues the same conversation
thread_config = {"configurable": {"thread_id": "1"}}
 
# --- Turn 1: A complete question ---
response1 = agent.invoke(
    {"messages": [{"role": "user", "content": "What is your refund policy?"}]},
    thread_config,
)
print("Q: What is your refund policy?")
print("A:", response1["messages"][-1].content)
 
# --- Turn 2: A follow-up that depends on Turn 1 ---
response2 = agent.invoke(
    {"messages": [{"role": "user", "content": "Does that apply to digital products too?"}]},
    thread_config,
)
print("\nQ: Does that apply to digital products too?")
print("A:", response2["messages"][-1].content)

Output:

Q: What is your refund policy?
A: All physical products may be returned within 30 days of purchase for a full refund.
The original receipt or order confirmation email is required, and items must be in
their original packaging and unused condition.
After 30 days, returns are accepted for store credit only.
 
Q: Does that apply to digital products too?
A: Digital products (software licenses, e-books, online courses) are non-refundable
once the download or access link has been activated.
However, if you experience technical issues preventing access, you can contact support
within 7 days for a replacement or refund.

In the second turn, we passed "Does that apply to digital products too?" but the agent recognized from the conversation history that this was a follow-up about the refund policy, and accurately retrieved the digital product section from the refund policy documents.

Wait — we never added a query-rewriting step like the one from Section 11.1, so how did the agent handle the follow-up correctly? When the LLM calls a tool (a function decorated with @tool), it generates the tool's arguments itself. That includes the user query passed to retrieve_context — because the LLM has the full conversation history in front of it, it rewrote the follow-up into a complete, self-contained question before making the call. We never set up a dedicated rewriting step, yet query rewriting still happened as part of the tool-calling process.

Note, too, that we never had to manage the conversation history ourselves — InMemorySaver handles it automatically for each thread_id.

The next section covers the problem that arises as conversations grow longer and the history gets larger, along with how to solve it.

11.3) Managing Longer Conversations

The conversational RAG we built works well at first, but problems can arise as conversations grow longer. As we learned in Chapter 8, LLMs have a maximum input size they can process in a single call. The system prompt, conversation history, retrieved documents, and user question all need to fit within this limit.

As conversations grow longer, the conversation history takes up more tokens, eventually exceeding the maximum input size and causing API calls to fail. Costs also increase with each call since you're billed per token. This means we need to manage the size of our conversation history.

In Chapter 8 we solved this problem with a sliding window: keeping only the most recent N messages and discarding older ones. The same concept applies in an agent environment. create_agent supports middleware, which is a processing step that can modify messages before the LLM is called. We can use middleware to trim old history.

11.3.1) Limiting History with Middleware

The @before_model decorator works similarly to the @tool decorator we saw in Section 11.2. Just as @tool converts a function into a tool the agent can use, @before_model converts a function into middleware that runs before each LLM call. The converted middleware is activated by registering it in the middleware parameter of create_agent.

from langchain.agents import create_agent, AgentState
from langchain.agents.middleware import before_model
from langchain.messages import RemoveMessage
from langgraph.graph.message import REMOVE_ALL_MESSAGES
 
@before_model
def trim_old_messages(state: AgentState, runtime) -> dict | None:
    """Remove old messages before each LLM call."""
    messages = state["messages"]
    # If there are few enough messages, do nothing
    if len(messages) <= 10:
        return None
    # Keep only the system message (first) and the 10 most recent messages
    return {
        "messages": [
            RemoveMessage(id=REMOVE_ALL_MESSAGES),
            messages[0],     # System message
            *messages[-10:], # Last 10 messages (5 turns)
        ]
    }

AgentState is an object that holds the agent's state data, with state["messages"] containing the list of conversation messages so far. The middleware's return value determines how this conversation list is modified.

• Returning None leaves the existing agent state data unchanged.
• Returning a dictionary applies its contents to the existing message list. In the code above, RemoveMessage(id=REMOVE_ALL_MESSAGES) first deletes all existing messages, then adds back only the system message and the 10 most recent messages. As a result, only these messages are passed to the LLM.

Register this middleware with the agent:

agent = create_agent(
    model="gpt-5-mini",
    tools=[retrieve_context],
    system_prompt=(
        "You are a helpful assistant that answers questions about company policies. "
        "Use the retrieve_context tool to search for relevant information. "
        "If the retrieved context does not contain relevant information, "
        "say that you don't know. "
        "Keep the answer concise, three sentences maximum."
    ),
    checkpointer=InMemorySaver(),
    middleware=[trim_old_messages],  # Register middleware
)

This is the same agent from Section 11.2 with middleware=[trim_old_messages] added. Now, no matter how long the conversation gets, only recent messages are passed to the LLM.

11.3.2) The Sliding Window Trade-off

When old messages are trimmed, the agent can no longer reference their content. If a user brings up something they asked ten turns ago, the agent has no way of knowing that context. This is a fundamental limitation of the sliding window approach.

When older conversation content needs to be preserved, an alternative is to replace old messages with an LLM-generated summary instead of deleting them. LangChain provides SummarizationMiddleware for this purpose, which we'll cover in Part V (Chapter 15 onward) when we dive into agent and graph architectures.