9. Build Your First RAG System

Every app we've built so far relied only on the LLM's pre-trained knowledge. That's why it couldn't answer questions about information the LLM never learned—like your company's internal documents or product manuals.

RAG (Retrieval-Augmented Generation) solves this problem. When a user question comes in, it first retrieves relevant documents, then passes the retrieved content along with the question to the LLM so it can answer based on that content. You're combining the LLM's reasoning ability with your document knowledge.

In this chapter, we'll build a complete RAG pipeline from document preparation (loading, chunking, embedding) to retrieval-based answer generation. The finished system retrieves relevant documents when a question comes in, then passes them along with the question to the LLM so it answers based on that document content. It answers accurately when the information is in the documents, and honestly says "I don't know" when it's not—this is the essence of trustworthy RAG.

9.1) Understanding RAG

9.1.1) The Problem: LLMs Don't Know Your Data

LLMs are trained on internet data like Wikipedia, news articles, and public code. They don't know about your company's internal documents or the contract you received yesterday. So they can't answer questions like:

• "What's our company vacation policy?"
• "Summarize this quarter's sales report"
• "What are the refund terms in the contract I just received?"

from langchain_openai import ChatOpenAI
 
llm = ChatOpenAI(model="gpt-5-mini")
 
# Ask about a private document the LLM has never seen
response = llm.invoke("What is the refund policy for Acme Corp?")
print(response.content)

Output:

I don't have specific information about Acme Corp's refund policy. I'd recommend
checking their official website or contacting their customer support team directly
for the most accurate and up-to-date information.

In this example, the LLM honestly says it doesn't know. (Or it might hallucinate a plausible-sounding answer.)

But what if we provide the refund policy document along with the question? The LLM would give an accurate answer based on the provided content. This is the core idea behind RAG.

9.1.2) How Should We Provide the Document?

The simplest approach is to copy and paste the entire document into the prompt. This actually works well for short documents.

from langchain_openai import ChatOpenAI
 
llm = ChatOpenAI(model="gpt-5-mini")
 
# In reality, this would be much longer, but let's assume the following is the entire document
document_text = """
Refund Policy (Effective January 2026):
- Full refund within 30 days of purchase with original receipt.
- After 30 days, store credit only.
- Digital products are non-refundable after download.
- Defective items may be returned at any time for full refund.
"""
 
response = llm.invoke(
    f"""Please answer based on the following document:
{document_text}
 
Question: What is the refund policy for digital products?"""
)
print(response.content)

Output:

Digital products are non-refundable after download.
...

This works well for short documents. But what if the document is very large? This causes the following serious problems:

1. Context Window Limits: LLMs have a limited number of tokens they can process at once. For GPT-5-mini, it's 400K tokens. However, your entire company documentation can easily exceed this. Even if it fits, responses become slower and less accurate as the context grows longer.

2. Cost: LLM APIs charge per token. Sending the entire document when only one or two paragraphs are needed makes costs skyrocket.

3. Accuracy Degradation: When you include the entire document, the information you actually need gets buried in irrelevant content. The LLM's attention gets diverted by unrelated information, degrading answer quality.

RAG solves all three problems by retrieving and providing only the relevant parts of the document.

9.1.3) Core Idea: Retrieve Relevant Parts and Provide Them with the Question

The essence of RAG is simple: Before sending the question to the LLM, first find the relevant parts of your documents and provide them together with the question.

Here's how it works:

1. User asks a question.
2. The system retrieves (Retrieval) relevant content from the document store.
3. The retrieved content is added (Augmentation) to the prompt along with the question.
4. The LLM generates (Generation) an answer based on the retrieved content.

These three steps are where RAG (Retrieval-Augmented Generation) gets its name.

9.1.4) How Do We Retrieve Relevant Content? (Limitations of Keyword Matching)

The retrieval step is crucial to RAG. You need to provide relevant content to get proper answers. So how do we retrieve content related to the question?

The simplest method is keyword matching: finding documents that contain words from the question. For example, if someone asks "What's the refund policy for digital products?" you'd search for documents containing the words "refund," "digital," and "products."

But keyword matching has a critical weakness: it can only find exact word matches.

Let's say you have a refund policy document with this content:

"Full refund available within 30 days of purchase."

What happens when a user asks "How do I get my money back?" This document won't be retrieved. The document doesn't contain the phrase "get my money back." Humans understand that "refund" and "get my money back" have the same meaning, but keyword search only matches words, so it fails to find it.

Keyword search only matches words. Even when the meaning is the same, if the words differ, it won't find it.

The solution is semantic search. And what makes that possible is embeddings.

9.1.5) Embeddings: Converting Text into Numerical Vectors

Embeddings represent the meaning of text as a list of numbers (a vector). When you input text into an embedding model, it converts it into a vector of hundreds to thousands of numbers.

from langchain_openai import OpenAIEmbeddings
 
embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")
 
# Embed a single sentence
vector = embeddings_model.embed_query("How do I get my money back?")
 
print(f"Vector dimensions: {len(vector)}")
print(f"First 5 values: {vector[:5]}")

Output:

Vector dimensions: 1536
First 5 values: [0.0123, -0.0456, 0.0789, -0.0234, 0.0567]

Dimension is the number of values that make up the vector. The text-embedding-3-small model represents all text as 1,536 numbers.

Why do we need so many numbers? Because each dimension captures different aspects of meaning:

• Some dimensions might distinguish "action/state"
• Others might represent degrees of "concrete/abstract"
• Still others might indicate "positive/negative" sentiment
• ... (1,536 semantic features—though we can't actually interpret what each dimension represents)

Just as 2D coordinates (x, y) represent a point on a plane, a 1,536-dimensional vector represents a point in a 1,536-dimensional "meaning space." More dimensions allow finer distinctions in meaning.

Similar meanings are located close together in meaning space. "Refund method" and "get money back" use different words, but because they have similar meanings, they're placed close together in meaning space.

9.1.6) Semantic Search: Similar Meaning, Closer Distance

Once you've converted both documents and queries into vectors, you can find the most relevant documents by measuring similarity between vectors. This is called semantic search — searching by semantic similarity rather than keyword matching.

The most common similarity measure is cosine similarity, which measures the angle between two vectors. When vectors point in similar directions, similarity is higher. Closer to 1.0 means very similar meaning, while closer to 0 means low relevance.

Let's actually compute this:

from langchain_openai import OpenAIEmbeddings
import numpy as np
 
embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")
 
# Embed the query and two candidate documents
query_vec = embeddings_model.embed_query("What's the refund period?")
doc1_vec = embeddings_model.embed_query("Full refund available within 30 days of purchase.")  # Related
doc2_vec = embeddings_model.embed_query("Our office is located in downtown Seattle.")  # Unrelated
 
def cosine_similarity(a, b):
    """Compute cosine similarity between two vectors."""
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
 
sim1 = cosine_similarity(query_vec, doc1_vec)
sim2 = cosine_similarity(query_vec, doc2_vec)
 
print(f"Query vs 'refund' doc:    {sim1:.4f}")
print(f"Query vs 'office' doc:   {sim2:.4f}")

Output:

Query vs 'refund' doc:    0.6415
Query vs 'office' doc:   0.1706

(Actual values may vary by model)

The refund document scores much higher. The embedding model understands that "refund period" and "full refund within 30 days" are semantically related. This is semantic search, and it's the core mechanism of RAG.

9.1.7) RAG Pipeline Overview

Combining the concepts we've learned creates the following RAG pipeline:

The pipeline consists of two phases:

Knowledge Ingestion (performed once initially, or when documents change):

1. Document Loading: Extract text data from various sources (Markdown, PDF, etc.).
2. Text Splitting (Chunking): Split long documents into smaller chunks to improve retrieval precision and comply with LLM input limits.
3. Vector Conversion (Embedding): Use an embedding model to convert chunks into meaning-based numerical vectors.
4. Vector Storage: Store the converted vectors and original text in a vector database (indexing).

Retrieval and Answer Generation (performed for each user question):

1. Question Embedding: Convert the user's question into a numerical vector using the same model used during ingestion.
2. Similarity Search (Retrieval): Extract the top-K chunks that are semantically closest to the question vector from the vector database.
3. Prompt Augmentation: Combine the original question with the retrieved chunks to augment the prompt.
4. Answer Generation: The LLM references the provided chunks to generate a grounded answer.

9.2) Document Loading and Chunking

This section covers the first two steps of the RAG pipeline's knowledge ingestion phase:

1. Document Loading: Reading text data from files
2. Text Splitting (Chunking): Breaking text data into small, searchable pieces

In the next section (9.3), we will learn how to convert these chunks into vectors and store them.

9.2.1) Loading Documents from Files

The first step in a RAG pipeline is loading documents into Python objects. LangChain provides document loaders — classes that support a variety of file formats. The main loaders are:

• TextLoader: Plain text (.txt) and Markdown (.md) files
• PyPDFLoader: PDF (.pdf) files, loaded page by page
• CSVLoader: CSV (.csv) files, with each row loaded as a separate document
• UnstructuredMarkdownLoader: Markdown (.md) files, with structural awareness (headers, lists, etc.)

Regardless of which loader you use, the result is always returned as a list of Document objects. Each Document has two key attributes:

• page_content: The text content of the document
• metadata: A dictionary containing meta-information such as file path and page number

In this tutorial, we will use TextLoader to load Markdown files.

Preparing Sample Documents

First, let's create some sample documents to work with. Create a data/docs/ folder in your project and add the following files:

mkdir -p data/docs

Create data/docs/refund_policy.md:

# Refund Policy
 
**Effective Date**: January 1, 2026
 
## Standard Returns
 
All physical products may be returned within 30 days of purchase for a full refund.
The original receipt or order confirmation email is required. Items must be in their
original packaging and unused condition.
 
After 30 days, returns are accepted for store credit only. Store credit does not expire.
 
## Digital Products
 
Digital products (software licenses, e-books, online courses) are non-refundable
once the download or access link has been activated. If you experience technical
issues preventing access, contact support within 7 days for a replacement or refund.
 
## Defective Items
 
Defective items may be returned at any time for a full refund or replacement.
Please include a description of the defect. Shipping costs for defective returns
are covered by the company.
 
## Subscription Services
 
Monthly subscriptions may be cancelled at any time. Refunds are prorated based on
the remaining days in the billing cycle. Annual subscriptions may be refunded in full
within the first 14 days. After 14 days, no refund is available but access continues until the end of the billing period.

Create data/docs/shipping_info.md:

# Shipping Information
 
## Domestic Shipping
 
Standard shipping (5-7 business days): Free on orders over $50, otherwise $5.99.
Express shipping (2-3 business days): $12.99.
Overnight shipping (next business day): $24.99.
 
## International Shipping
 
International orders are shipped via tracked airmail. Delivery times vary by
destination, typically 10-21 business days. International shipping costs are
calculated at checkout based on weight and destination.
 
Customs duties and import taxes are the responsibility of the buyer and are not included in the shipping cost.
 
## Order Tracking
 
All orders include a tracking number sent via email within 24 hours of shipment.
Track your order through the tracking link in your email or through the carrier's website.
 
## Lost or Damaged Packages
 
If your package is lost or arrives damaged, contact support within 48 hours.
We will ship a replacement at no additional cost. For damaged items, please
provide photos of the damage and packaging.

Now load these files using TextLoader:

from pathlib import Path
from langchain_community.document_loaders import TextLoader
 
# Load all .md files from the data/docs directory
docs_dir = Path("data/docs")
 
for md_file in docs_dir.glob("*.md"):
    loader = TextLoader(str(md_file), encoding="utf-8")
    docs = loader.load()
 
    if docs:  # Check that the file is not empty
        doc = docs[0]  # Single file = single Document
        print(f"File: {doc.metadata['source']}")
        print(f"Length: {len(doc.page_content)} characters")
        print(f"Preview: {doc.page_content[:80]}...")
        print()

Output:

File: data/docs/refund_policy.md
Length: 1166 characters
Preview: # Refund Policy
...
 
File: data/docs/shipping_info.md
Length: 972 characters
Preview: # Shipping Information
...

Note: TextLoader takes a single file path as input, but returns List[Document] for a consistent interface with other loaders. (For example, PDFLoader returns multiple Documents — one per page.)

9.2.2) Splitting Documents into Chunks: Chunking

The two documents above are intentionally short for the sake of this tutorial. In real applications, you will often work with documents that are hundreds or thousands of pages long. If you embed an entire document as a single vector, thousands of concepts get compressed into one — making it impossible to accurately retrieve what you actually need.

Chunking is the process of splitting documents into small, meaningful pieces. The goal is simple: when a user asks a question, only the specific paragraphs directly relevant to the answer should be retrieved — not the entire document.

Chunk size directly affects both retrieval and answer quality:

• Too large: Multiple topics get mixed into one chunk, making embeddings less accurate and retrieval harder. Even when the right chunk is found, irrelevant content gets passed to the LLM, degrading answer quality.
• Too small: The LLM may not receive enough information to answer correctly. For example, if only the sentence "Standard shipping is $5.99" is retrieved, the LLM cannot know that this only applies to orders under $50.
• Just right: Each chunk covers one topic with enough context, enabling accurate retrieval and answers.

9.2.3) Controlling Chunk Size and Overlap

To split documents into chunks, you need a text splitter. A text splitter is a LangChain tool that breaks long documents into smaller pieces. Choosing the right splitter is important.

• RecursiveCharacterTextSplitter: Tries multiple separators in hierarchical order to preserve as much context as possible. The most widely used splitter for general purposes.
• CharacterTextSplitter: Splits on a single separator (default: \n\n). Suitable for documents with simple structure.
• MarkdownHeaderTextSplitter: Splits on Markdown headers (#, ##). Effective when you want to preserve the document's table of contents structure.

Why is RecursiveCharacterTextSplitter effective?

This splitter works by trying separators from largest to smallest unit to find the best split point. The default order is as follows (can be changed via the separators parameter):

paragraph (\n\n) → line break (\n) → word ( )

It always tries to split at the largest meaningful unit first. If a paragraph exceeds chunk_size, it falls back to line breaks, then words. Because it always finds the most natural split point rather than cutting arbitrarily in the middle of a word, the resulting chunks are more likely to contain semantically complete information.

Key Parameters

• chunk_size: The maximum number of characters per chunk. For example, chunk_size=400 means no chunk will exceed 400 characters.
• chunk_overlap: The number of overlapping characters between adjacent chunks. For example, chunk_overlap=80 means the last 80 characters of one chunk are repeated at the start of the next.
• separators: The list of separators used to split the text, tried in priority order. If splitting on the current separator would exceed chunk_size, the next separator is tried to avoid exceeding chunk_size.

What is overlap and why is it needed?

Overlap means adjacent chunks share some content — the end of one chunk is included at the start of the next.

The reason for this is to ensure that each chunk can stand on its own with sufficient context. When reading a piece of a document without any knowledge of what came before, it can be hard to understand why certain content is being mentioned. Overlap keeps the end of one chunk flowing into the next, so that whichever chunk is retrieved, the content reads naturally.

Now let's actually split a document:

from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
 
# Load the document
loader = TextLoader("data/docs/refund_policy.md", encoding="utf-8")
docs = loader.load()
 
# Configure the splitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,
    chunk_overlap=80,
    separators=["\n## ", "\n\n", "\n", " "],
)
 
chunks = text_splitter.split_documents(docs)
 
print(f"Split into {len(chunks)} chunks\n")
for i, chunk in enumerate(chunks):
    print(f"--- Chunk {i} (source: {chunk.metadata['source']}) ---")
    print(f"Length: {len(chunk.page_content)} characters")
    print(chunk.page_content[:120])
    print()

Output:

Split into 4 chunks
 
--- Chunk 0 (source: data/docs/refund_policy.md) ---
Length: 374 characters
# Refund Policy
...
 
--- Chunk 1 (source: data/docs/refund_policy.md) ---
Length: 267 characters
## Digital Products
...
 
--- Chunk 2 (source: data/docs/refund_policy.md) ---
Length: 204 characters
## Defective Items
...
 
--- Chunk 3 (source: data/docs/refund_policy.md) ---
Length: 315 characters
## Subscription Services
...

Note: In the example above, no overlap occurred. This is because each paragraph was cleanly split based on the first separator (\n## ) while staying within the chunk_size. Overlap only occurs when a specific paragraph is longer than the chunk_size and must be divided into two or more pieces.

9.3) Vector Storage and Retrieval with ChromaDB

9.3.1) What Is a Vector Store?

A vector store (also called a vector database) is a database optimized for storing and searching data using embedding vectors. Unlike a traditional database where you query by exact field values (SELECT * FROM products WHERE category = 'electronics'), a vector store finds the items with the most similar meaning to your query.

In RAG, the vector store holds document chunks along with their embeddings. When a user asks a question, the question is converted into a vector, and the vector store retrieves the chunks with the most similar vectors.

9.3.2) Choosing a Vector Store and Setting Up ChromaDB

Popular vector stores include ChromaDB, Pinecone, Weaviate, and pgvector (PostgreSQL extension). They differ in hosting model (local vs. cloud), scale, and operational complexity. For this book, we will use ChromaDB — it is open-source, runs entirely on your local machine with no server setup, and is useful not just for development but also for small-to-medium production workloads.

ChromaDB can be used in several ways:

• Local mode (pip): Install it as a Python library and use it immediately. You can store and load data in a local directory without any separate server infrastructure.
• Standalone server (Docker): Run ChromaDB as a separate server process. Useful when multiple applications need to share the same vector store.
• Managed cloud service (Chroma Cloud): Use ChromaDB as a cloud service. Chroma Cloud handles hosting, scaling, and maintenance, allowing you to deliver a stable service without infrastructure management burden.

Let's install ChromaDB using pip:

pip install chromadb langchain-chroma

chromadb is the core vector store library, and langchain-chroma is an integration package that allows you to use ChromaDB directly within the LangChain library.

9.3.3) Choosing the Embedding Model

The first thing to decide is which embedding model to use. Embedding vectors can only be compared when they are generated by the same model. Therefore, you must use the same embedding model for both storing documents and querying.

OpenAI provides the following embedding models:

For this book, we will use OpenAI's text-embedding-3-small model. It offers high efficiency at a low cost, making it a practical choice for general search, RAG, and cost-conscious projects.

from langchain_openai import OpenAIEmbeddings
 
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
 
# Verify it works
test_vector = embedding_model.embed_query("test")
print(f"Embedding dimensions: {len(test_vector)}")

Output:

Embedding dimensions: 1536

Cost note: Embedding API calls are much cheaper than LLM calls, but they do incur costs. When storing documents in the database (indexing), one API call is required per chunk, and when a user asks a question (retrieval), one API call is required for the question. For current pricing, refer to the OpenAI pricing page.

9.3.4) Storing Chunks in ChromaDB

Now let's put everything together. We will load documents, split them into chunks, embed the chunks, and store them along with their embedding vectors in ChromaDB.

# ingest.py - Complete ingestion pipeline
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
 
# Step 1: Load documents
# DirectoryLoader: scans a directory and loads matching files.
# The actual loading is delegated to the loader specified in loader_cls.
loader = DirectoryLoader(
    "data/docs/", glob="**/*.md",
    loader_cls=TextLoader, loader_kwargs={"encoding": "utf-8"},
)
documents = loader.load()
print(f"Loaded {len(documents)} documents")
 
# Step 2: Split into chunks
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=400,
    chunk_overlap=80,
    separators=["\n## ", "\n\n", "\n", " ", ""],
)
chunks = text_splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks")
 
# Step 3: Create embedding model
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
 
# Step 4: Create vector store and ingest chunks
vector_store = Chroma.from_documents(
    documents=chunks,
    embedding=embedding_model,
    persist_directory="data/chroma_db",
    collection_name="company_docs",
)
 
print(f"Stored {len(chunks)} chunks in ChromaDB at data/chroma_db/")

Output:

Loaded 2 documents
Created 8 chunks
Stored 8 chunks in ChromaDB at data/chroma_db/

The Chroma.from_documents() method performs two tasks in a single call:

1. Passes the chunks provided via the documents parameter through the embedding model to obtain embedding vectors.
2. Stores each chunk together with its embedding vector in ChromaDB.

9.3.5) Loading a Persisted Vector Store

In the previous section, we stored documents in the vector store. This storage operation only needs to be performed once initially (or when documents change). After that, you can simply load the persisted vector store and use it directly.

from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
 
# Load a persisted vector store — no re-embedding needed
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
 
vector_store = Chroma(
    persist_directory="data/chroma_db",
    collection_name="company_docs",
    embedding_function=embedding_model,
)
 
print(f"Loaded vector store with {len(vector_store.get()['ids'])} chunks")

Output:

Loaded vector store with 8 chunks

Now you can start searching immediately by simply loading the persisted vector store, without needing to re-embed your documents.

9.3.6) Similarity Search

With the vector store loaded, you can now search for chunks that are semantically similar to a query. The top-K parameter specifies how many results to return:

from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
 
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = Chroma(
    persist_directory="data/chroma_db",
    collection_name="company_docs",
    embedding_function=embedding_model,
)
 
# Search for chunks related to a question
query = "Can I return a digital product?"
results = vector_store.similarity_search(query, k=2)
 
print(f"Query: {query}")
print(f"Found {len(results)} results\n")
 
for i, doc in enumerate(results):
    print(f"--- Result {i + 1} (source: {doc.metadata['source']}) ---")
    print(doc.page_content[:200])
    print()

Output:

Query: Can I return a digital product?
Found 2 results
 
--- Result 1 (source: data/docs/refund_policy.md) ---
## Digital Products
 
...
 
--- Result 2 (source: data/docs/refund_policy.md) ---
# Refund Policy
 
**Effective Date**: January 1, 2026
 
## Standard Returns
 
...

For this query, the digital products chunk was retrieved with the highest similarity, followed by the refund policy chunk.

You can also retrieve results with their similarity scores using similarity_search_with_score:

results_with_scores = vector_store.similarity_search_with_score(query, k=2)
 
for doc, score in results_with_scores:
    # ChromaDB returns distance (lower = more similar)
    print(f"Score: {score:.4f} | Source: {doc.metadata['source']}")
    print(f"  {doc.page_content[:200]}...")
    print()

Output:

Score: 0.5942 | Source: data/docs/refund_policy.md
  ## Digital Products
 
...
 
Score: 0.9577 | Source: data/docs/refund_policy.md
  # Refund Policy
 
...

Note that ChromaDB uses distance scores (lower is more similar), not similarity scores (higher is more similar). The digital products chunk has the lowest distance of 0.5942, making it the most relevant result.

9.4) Building the Complete RAG Chain

Now we'll build a complete RAG system: retrieve relevant documents first, then pass them along with the question to the LLM to generate answers based on the provided information.

9.4.1) Designing the Prompt Template

The most important part of the prompt template is instructing the LLM to answer based only on the provided context. Without this instruction, the LLM may ignore the search results and fabricate answers based on its training data.

from langchain_core.prompts import ChatPromptTemplate
 
rag_prompt = ChatPromptTemplate.from_messages([
    ("system",
     "You are a customer service representative. "
     "Answer the user's question using ONLY the provided context. "
     "If the context does not contain enough information to answer, "
     "say \"I don't have enough information to answer that question.\"\n\n"
     "Context:\n{context}"),
    ("human", "{question}"),
])

The system message forces the LLM to answer using only the provided context. Crucially, the instruction to say "I don't have enough information" when the context is insufficient prevents the LLM from fabricating plausible but unsupported answers.

9.4.2) Building the RAG Chain

We now have all the components ready. We just need to connect the retriever, prompt template, and LLM.

The completed RAG system will work as follows:

1. Receive the user's question
2. Retrieve relevant chunks from the vector store
3. Pass the chunks and question to the prompt template to generate the prompt
4. Generate an answer with the LLM

Let's connect the RAG chain using the LCEL | operator from Chapter 6.

# rag_chain.py - Complete RAG pipeline
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_chroma import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
 
 
def format_docs(docs):
    """Join retrieved documents into a single context string."""
    return "\n\n---\n\n".join(doc.page_content for doc in docs)
 
 
def build_rag_chain():
    """Build and return the complete RAG chain."""
    # Load the vector store
    embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
    vector_store = Chroma(
        persist_directory="data/chroma_db",
        collection_name="company_docs",
        embedding_function=embedding_model,
    )
 
    # Create a retriever (k=3 means return top 3 chunks)
    retriever = vector_store.as_retriever(search_kwargs={"k": 3})
 
    # Define the prompt
    rag_prompt = ChatPromptTemplate.from_messages([
        ("system",
         "You are a customer service representative. "
         "Answer the user's question using ONLY the provided context. "
         "If the context does not contain enough information to answer, "
         "say \"I don't have enough information to answer that question.\"\n\n"
         "Context:\n{context}"),
        ("human", "{question}"),
    ])
 
    # Initialize the LLM
    llm = ChatOpenAI(model="gpt-5-mini")
 
    # Compose the chain using LCEL
    rag_chain = (
        {"context": retriever | format_docs, "question": lambda x: x}
        | rag_prompt
        | llm
        | StrOutputParser()
    )
 
    return rag_chain
 
 
if __name__ == "__main__":
    chain = build_rag_chain()
    answer = chain.invoke("Can I return a digital product?")
    print(answer)

Output:

Digital products are non-refundable once the download or access link has been activated.
If you experience technical issues preventing access, contact support within 7 days for a replacement or refund.

Let's break down the chain composition step by step:

rag_chain = (
    {"context": retriever | format_docs, "question": lambda x: x}
    | rag_prompt
    | llm
    | StrOutputParser()
)

When you call chain.invoke("Can I return a digital product?"), here's what happens:

1. Dictionary step:
   • retriever | format_docs: Searches the vector store with the question and combines the chunks into a single string
   • lambda x: x: Passes the question through unchanged
   • Result: {"context": "retrieved chunks (combined into a single string)", "question": "Can I return a digital product?"}
2. rag_prompt: Fills the {context} and {question} placeholders in the prompt template with the dictionary values
3. llm: Sends the completed prompt to the LLM
4. StrOutputParser(): Extracts just the text from the LLM's response

For more details on how LCEL works, see Chapter 6.

9.4.3) Testing with Answerable and Unanswerable Questions

A RAG system must handle both questions it can answer (information exists in the documents) and questions it cannot answer (information is not in the documents). Let's test both scenarios:

# test_rag.py - Test the RAG chain with various questions
from rag_chain import build_rag_chain
 
chain = build_rag_chain()
 
test_questions = [
    # Answerable — information is in the documents
    "What is the refund policy for physical products?",
    "How much does express shipping cost?",
    "Can I return a defective item after 6 months?",
    # Unanswerable — information is NOT in the documents
    "What is the employee vacation policy?",
    "What programming languages are used?",
]
 
for question in test_questions:
    print(f"Q: {question}")
    answer = chain.invoke(question)
    print(f"A: {answer}\n")
    print("-" * 60)

Output:

Q: What is the refund policy for physical products?
A: All physical products may be returned within 30 days of purchase for a full refund. ...
 
------------------------------------------------------------
Q: How much does express shipping cost?
A: Express shipping (2–3 business days) costs $12.99.
 
------------------------------------------------------------
Q: Can I return a defective item after 6 months?
A: Yes. Defective items may be returned at any time for a full refund or replacement. ...
 
------------------------------------------------------------
Q: What is the employee vacation policy?
A: I don't have enough information to answer that question.
 
------------------------------------------------------------
Q: What programming languages are used?
A: I don't have enough information to answer that question.
 
------------------------------------------------------------

The results demonstrate exactly the behavior we want:

• Answerable questions: Provide accurate answers based on the retrieved documents. The LLM doesn't add information not contained in the documents.
• Unanswerable questions: Respond with "I don't have enough information to answer that question." The LLM correctly identifies that the retrieved context lacks relevant information and refuses to fabricate an answer.

This is the power of RAG. Your LLM answers questions about your data and honestly admits when it doesn't know. Every answer is backed by documents, making the system far more trustworthy than a standard LLM.