10. Smarter Retrieval: Filters, Thresholds, and MMR

In Chapter 9, we built a simple RAG system. It was a pipeline that split documents into chunks, embedded them into ChromaDB, and when a user question came in, retrieved relevant chunks and included them in the prompt. This allowed the LLM to answer questions about information it was never trained on, like internal company documents and product manuals. Everything seemed to work just fine.

But try asking a wider variety of questions and weaknesses emerge quickly. You ask about the consumer refund policy and employee store content gets mixed in, or you ask about something that's not in any document and the LLM fabricates a plausible answer, or you increase the number of search results and answer quality actually drops.

In this chapter, we'll address these three problems one by one. We'll use metadata filtering to restrict which documents are searched, similarity score thresholds to exclude results unrelated to the question, and K and MMR tuning to improve the quantity and diversity of chunks fed to the LLM. No new tools are needed. We're refining the configuration and usage of similarity_search() and the Chroma vector store that you already know.

10.1) What's Wrong with Our RAG?

In Chapter 9, we only put two documents into the vector store — a refund policy and a shipping policy — and each document covered a distinct topic. We also only tested questions where answers were either clearly present or absent in the documents. This time, we'll create a more realistic scenario. We'll add an employee store guide to the vector store. This document also contains refund-related content, but it's meant for employees, not general consumers. Then we'll ask various questions and see what problems arise.

Data Preparation: Adding the Employee Store Guide

Here are the two documents from Chapter 9 for reference.

data/docs/refund_policy.md:

markdown

# Refund Policy
 
**Effective Date**: January 1, 2026
 
## Standard Returns
 
All physical products may be returned within 30 days of purchase for a full refund.
The original receipt or order confirmation email is required. Items must be in their
original packaging and unused condition.
 
After 30 days, returns are accepted for store credit only. Store credit does not expire.
 
## Digital Products
 
Digital products (software licenses, e-books, online courses) are non-refundable
once the download or access link has been activated. If you experience technical
issues preventing access, contact support within 7 days for a replacement or refund.
 
## Defective Items
 
Defective items may be returned at any time for a full refund or replacement.
Please include a description of the defect. Shipping costs for defective returns
are covered by the company.
 
## Subscription Services
 
Monthly subscriptions may be cancelled at any time. Refunds are prorated based on
the remaining days in the billing cycle. Annual subscriptions may be refunded in full
within the first 14 days. After 14 days, no refund is available but access continues until the end of the billing period.

data/docs/shipping_info.md:

markdown

# Shipping Information
 
## Domestic Shipping
 
Standard shipping (5-7 business days): Free on orders over $50, otherwise $5.99.
Express shipping (2-3 business days): $12.99.
Overnight shipping (next business day): $24.99.
 
## International Shipping
 
International orders are shipped via tracked airmail. Delivery times vary by
destination, typically 10-21 business days. International shipping costs are
calculated at checkout based on weight and destination.
 
Customs duties and import taxes are the responsibility of the buyer and are not included in the shipping cost.
 
## Order Tracking
 
All orders include a tracking number sent via email within 24 hours of shipment.
Track your order through the tracking link in your email or through the carrier's website.
 
## Lost or Damaged Packages
 
If your package is lost or arrives damaged, contact support within 48 hours.
We will ship a replacement at no additional cost. For damaged items, please
provide photos of the damage and packaging.

Add the employee store guide here.

Create data/docs/employee_store.md:

markdown

# Employee Store Guide
 
## Eligibility and Benefits
 
Employees can purchase company products at a 30% discount through the internal employee store.
The monthly purchase limit is $500, and payment can be made via payroll deduction or benefit points.
 
## Ordering and Shipping
 
Employee store orders are placed through the internal portal, and delivery is only available to the company address.
Orders are delivered within 3-5 business days, and shipping is free.
 
## Refund Policy
 
Refunds are available within 7 days of purchase for unopened items only.
Cash refunds are not available; refunds are credited as benefit points.
After opening, only exchanges are allowed, limited to one exchange per identical product.
 
## Contact
 
For employee store inquiries, please contact HR at hr@acme.com.

The data/docs/ directory now contains three files: refund_policy.md, shipping_info.md, employee_store.md. Re-run the ingestion script from Chapter 9 to rebuild the vector store.

python

# ingest.py — same ingestion pipeline as Chapter 9
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
 
loader = DirectoryLoader(
    "data/docs/", glob="**/*.md",
    loader_cls=TextLoader, loader_kwargs={"encoding": "utf-8"},
)
documents = loader.load()
print(f"Loaded {len(documents)} documents")
 
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=200,
    chunk_overlap=80,
    separators=["\n## ", "\n\n", "\n", " ", ""],
)
chunks = text_splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks")
 
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
 
vector_store = Chroma.from_documents(
    documents=chunks,
    embedding=embedding_model,
    persist_directory="data/chroma_db",
    collection_name="company_docs",
)
print(f"Stored {len(chunks)} chunks in ChromaDB")

Output:

Loaded 3 documents
Created 12 chunks
Stored 12 chunks in ChromaDB

Problem 1: Irrelevant Documents Mixed into Search Results

Let's search for refund conditions.

python

from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
 
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = Chroma(
    persist_directory="data/chroma_db",
    collection_name="company_docs",
    embedding_function=embedding_model,
)
 
results = vector_store.similarity_search("What are the refund conditions?", k=3)
 
for i, doc in enumerate(results):
    source = doc.metadata["source"]
    print(f"Result {i+1} [{source}] {doc.page_content[:80]}...")

Output:

Result 1 [data/docs/refund_policy.md] ## Standard Returns
All physical products may be returned within 30 days of pur...
 
Result 2 [data/docs/employee_store.md] ## Refund Policy
Refunds are available within 7 days of purchase for unopened i...
 
Result 3 [data/docs/refund_policy.md] ## Subscription Services
Monthly subscriptions may be cancelled at any time. Re...

Look at Result 2. A customer asked about refund conditions, and the employee store's refund policy is included in the results. The consumer refund policy allows full refunds within 30 days, but the employee store only allows refunds within 7 days for unopened items, with refunds credited as benefit points. If both policies are passed to the LLM together, the customer could be given employee-only refund terms.

Problem 2: Results Returned Even When No Relevant Content Exists

Now let's ask about something that doesn't exist anywhere in our documents. We'll use similarity_search_with_score(), which we learned in Chapter 9, to also see the distance values. In ChromaDB, lower distance values mean higher similarity.

python

results = vector_store.similarity_search_with_score(
    "What is the hiring process at this company?", k=3
)
 
for doc, score in results:
    source = doc.metadata["source"]
    print(f"[dist={score:.4f}] [{source}] {doc.page_content[:80]}...")

Output:

[dist=1.4648] [data/docs/employee_store.md] ## Refund Policy
Refunds are available within 7 days of purchase for unopened i...
 
[dist=1.4699] [data/docs/employee_store.md] ## Eligibility and Benefits
Employees can purchase company products at a 30% di...
 
[dist=1.4743] [data/docs/refund_policy.md] ## Subscription Services
Monthly subscriptions may be cancelled at any time. Re...

There is no information about the hiring process anywhere. The distance values are all above 1.4, showing very low similarity, yet similarity_search_with_score() still returned three chunks. Let's see what answer the RAG chain produces when these chunks are passed to it.

python

from rag_chain import build_rag_chain  # RAG chain from Chapter 9
 
chain = build_rag_chain()
answer = chain.invoke("What is the hiring process at this company?")
print(answer)

Output:

I don't have enough information to answer that question.

The LLM responded that it didn't have enough information to answer. This is because we included the instruction "say you don't have enough information" in build_rag_chain()'s system prompt. However, the LLM can't always make this judgment. If the retrieved chunks contain phrases that appear related to the question, the LLM may generate an incorrect answer based on that content.

Problem 3: Does Increasing Search Results Always Help?

You might think "wouldn't more context be better?" Increasing k from 3 to 10 does raise the chance that needed chunks are included. But at the same time, more irrelevant chunks come in too. Since the LLM receives all these chunks as context and generates answers from them, unnecessary or incorrect information can end up in the response. More context does not necessarily mean better answers.

Additionally, all retrieved chunks are passed to the LLM as tokens. As k grows, API call costs increase and response times slow down.

We've identified three problems. Now let's solve them one by one.

10.2) Metadata Filtering: Narrowing the Search Space

When we searched for refund conditions in Problem 1, both the customer refund policy and the employee store refund policy appeared together. This happened because we didn't tell similarity_search() which documents to search in.

Metadata filtering attaches attributes like category, source, and publication year to each chunk, then filters chunks based on these attributes before running the similarity search. Only chunks matching the conditions go through similarity computation. It serves a similar role to SQL's WHERE clause.

10.2.1) Document Content vs. Document Metadata

The Document object we learned about in Chapter 9 contains two things:

page_content: The text itself. It is embedded into a vector and is what similarity search operates on.
metadata: A dictionary that holds attributes like source and category. These values are not embedded.

Similarity search operates on page_content, while metadata filtering operates on metadata information.

10.2.2) Rebuilding the Vector Store: Adding Metadata

We'll add a category attribute for filtering and rebuild the vector store. We only need to add metadata assignment code to the ingest.py from 10.1.

python

# ingest_with_metadata.py
from langchain_community.document_loaders import DirectoryLoader, TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma
 
# Step 1: Load documents (same as 10.1)
loader = DirectoryLoader(
    "data/docs/", glob="**/*.md",
    loader_cls=TextLoader, loader_kwargs={"encoding": "utf-8"},
)
documents = loader.load()
print(f"Loaded {len(documents)} documents")
 
# Step 2: [NEW] Assign category metadata based on filename
CATEGORY_MAP = {
    "refund_policy.md": "customer",
    "shipping_info.md": "customer",
    "employee_store.md": "employee",
}
for doc in documents:
    filename = doc.metadata["source"].split("/")[-1]
    doc.metadata["category"] = CATEGORY_MAP.get(filename, "unknown")
 
# Step 3: Split into chunks (same as 10.1)
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=200,
    chunk_overlap=80,
    separators=["\n## ", "\n\n", "\n", " ", ""],
)
chunks = text_splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks")
 
# Step 4: Build the vector store (same as 10.1)
embedding_model = OpenAIEmbeddings(model="text-embedding-3-small")
 
vector_store = Chroma.from_documents(
    documents=chunks,
    embedding=embedding_model,
    persist_directory="data/chroma_db",
    collection_name="company_docs",
)
print(f"Stored {len(chunks)} chunks in ChromaDB")

Output:

Loaded 3 documents
Created 12 chunks
Stored 12 chunks in ChromaDB

The only difference from the original ingest.py is the addition of category metadata to each document.

10.2.3) Applying Filters to Search

We can now use the filter parameter of similarity_search() to restrict which documents are searched. We'll search with the same query as Problem 1, but add a filter to search only customer-facing documents.

python

results = vector_store.similarity_search(
    "What are the refund conditions?",
    k=3,
    filter={"category": "customer"},
)
 
for i, doc in enumerate(results):
    source = doc.metadata["source"]
    category = doc.metadata["category"]
    print(f"Result {i+1} [{category}] [{source}] {doc.page_content[:80]}...")

Output:

Result 1 [customer] [data/docs/refund_policy.md] ## Standard Returns
All physical products may be returned within 30 days of pur...
 
Result 2 [customer] [data/docs/refund_policy.md] ## Subscription Services
Monthly subscriptions may be cancelled at any time. Re...
 
Result 3 [customer] [data/docs/refund_policy.md] ## Digital Products
Digital products (software licenses, e-books, online course...

The employee store refund policy that was mixed in during Problem 1 is no longer included. This is because we specified that only chunks with category equal to "customer" should be searched.

Combining Multiple Conditions

The example above used a single condition (category equals "customer"). When multiple conditions need to be applied simultaneously, you can combine them with logical operators like $and and $or.

python

# $and: only chunks satisfying ALL conditions
filter={
    "$and": [
        {"category": "customer"},
        {"source": "data/docs/refund_policy.md"},
    ]
}
 
# $or: chunks satisfying ANY condition
filter={
    "$or": [
        {"source": "data/docs/refund_policy.md"},
        {"source": "data/docs/shipping_info.md"},
    ]
}

Additional operators include $ne (not equal), $gt (greater than), and $lt (less than). See the ChromaDB documentation for the full list of operators.

10.3) Similarity Thresholds: Excluding Low-Relevance Results

In Problem 2, when we asked about the hiring process, chunks with distance values above 1.4 were returned. In ChromaDB, distance values this high indicate almost no relevance. Yet they were still retrieved. This is because similarity_search() always returns k results.

This problem can be solved by setting a distance threshold. Results farther than the threshold are not included in the context passed to the LLM.

So what threshold should we set? Let's first compare the distance values between questions that have relevant content in the documents and questions that don't.

10.3.1) Comparing Distances: Questions With and Without Relevant Content

python

queries = [
    "Can I cancel a subscription service?",           # relevant content exists
    "What is the hiring process at this company?",     # no relevant content
]
 
for query in queries:
    print(f"\nQuery: {query}")
    results = vector_store.similarity_search_with_score(query, k=1)
    for doc, score in results:
        print(f"  [dist={score:.4f}] {doc.page_content[:80]}...")

Output:

Query: Can I cancel a subscription service?
  [dist=0.7511] ## Subscription Services
Monthly subscriptions may be cancelled at any time. Re...
 
Query: What is the hiring process at this company?
  [dist=1.4648] ## Refund Policy
Refunds are available within 7 days of purchase for unopened i...

Questions with relevant content have distance values around 0.75, while questions without relevant content have values above 1.4. Test several questions like this, then choose a threshold that clearly separates the two cases. The right threshold can vary depending on the embedding model, the nature of your documents, and chunk size, so it's best to determine it by testing directly with your own data.

10.3.2) Filtering Search Results by Distance Threshold

Once a threshold is set, let's build a function that filters out results exceeding the threshold. Only chunks that survive the filter are included in the context passed to the LLM. If no chunks remain below the threshold, we skip the LLM call entirely and respond with "I don't have enough information to answer that question."

python

from langchain_openai import ChatOpenAI
 
llm = ChatOpenAI(model="gpt-5-mini")
 
def retrieve_or_abstain(query: str, max_distance: float = 1.0, k: int = 3):
    """Return only chunks below the distance threshold. Returns None if none pass."""
    scored = vector_store.similarity_search_with_score(query, k=k)
    good = [doc for doc, dist in scored if dist <= max_distance]
    return good or None
 
def safe_answer(query: str) -> str:
    docs = retrieve_or_abstain(query)
    if docs is None:
        return "I don't have enough information to answer that question."
 
    context = "\n\n".join(d.page_content for d in docs)
    prompt = (
        "Answer the question using ONLY the context below.\n\n"
        f"Context:\n{context}\n\nQuestion: {query}"
    )
    return llm.invoke(prompt).content

Let's test with the same questions from earlier.

python

# Question with relevant content
print(safe_answer("Can I cancel a subscription service?"))
print("---")
# Question without relevant content
print(safe_answer("What is the hiring process at this company?"))

Output:

Yes. Monthly subscriptions may be cancelled at any time, and refunds are prorated
based on the remaining days in the billing cycle. Annual subscriptions may be refunded
in full within the first 14 days...
---
I don't have enough information to answer that question.

10.4) K and MMR: Controlling the Quantity and Diversity of Search Results

In this section, we'll directly compare how search results change with different k values, and learn about MMR search, which improves the diversity of results.

10.4.1) What Happens When You Increase K?

In Problem 3, we said that increasing k also brings in more irrelevant chunks. Let's verify this. We'll search with k=10 and examine the distance values of each chunk.

python

results = vector_store.similarity_search_with_score(
    "What are the refund conditions?",
    k=10,
    filter={"category": "customer"},
)
 
for i, (doc, dist) in enumerate(results, 1):
    source = doc.metadata["source"].split("/")[-1]
    print(f"{i:>2}. [dist={dist:.4f}] [{source}] {doc.page_content[:80]}...")

Output:

 1. [dist=0.7798] [refund_policy.md] ## Standard Returns
**Effective Date**: January 1, 2026
All physical products ma...
 
 2. [dist=0.8203] [refund_policy.md] ## Defective Items
Defective items may be returned at any time for a full refun...
 
 3. [dist=0.9353] [refund_policy.md] ## Subscription Services
Monthly subscriptions may be cancelled at any time. Re...
 
 4. [dist=1.4249] [shipping_info.md] ## Lost or Damaged Packages
If your package is lost or arrives damaged, contact...
 
 5. [dist=1.5516] [shipping_info.md] ## Domestic Shipping
Standard shipping (5-7 business days): Free on orders over...
...

The top 3 results have distances below 1.0 and are all related to refunds. Starting from the 4th result, distances jump above 1.4, and chunks unrelated to refund conditions — such as shipping information — begin appearing. With k=10, all of these chunks are passed to the LLM.

The costs of increasing k are as follows:

Noise: Lower-ranked chunks may be completely unrelated to the question. When such chunks are included in the prompt, the LLM may include unnecessary or incorrect information in its answer.
Higher cost: More chunks mean more tokens sent to the LLM, increasing API call costs.
Slower responses: More tokens to process means longer response times.

10.4.2) MMR: Achieving Both Relevance and Diversity

In real-world environments, as the document collection grows, it's common for multiple chunks with similar content to emerge. The chunk_overlap setting from Chapter 9, which made adjacent chunks share some content, is also a source of duplication. In such cases, even searching with k=3 could return three chunks that are nearly identical.

MMR (Maximum Marginal Relevance) is a search method that prevents results from being skewed toward the same content. Standard similarity search returns the k chunks closest to the query, which can cause similar chunks to cluster at the top. MMR prioritizes chunks that are both relevant to the query and different from the results already selected.

Here's how it works:

Just like standard similarity search, it first retrieves fetch_k candidate chunks closest to the query.
It selects the chunk closest to the query as the first result.
From the remaining candidates, it selects the next chunk that is relevant to the query but different in content from chunks already selected.
Step 3 repeats until k chunks are selected.

The result is a set of chunks that maintain relevance while avoiding overlapping content.

python

results_mmr = vector_store.max_marginal_relevance_search(
    "What are the refund conditions?",
    k=3,
    fetch_k=10,
)
 
for i, doc in enumerate(results_mmr):
    print(f"{i+1}. {doc.page_content[:80]}...")

Output:

1. ## Standard Returns
All physical products may be returned within 30 days of pur...
 
2. ## Defective Items
Defective items may be returned at any time for a full refun...
 
3. ## Digital Products
Digital products (software licenses, e-books, online course...

With the current data, there isn't a big difference from standard search because the dataset is small. However, as documents grow to hundreds or thousands, similar chunks frequently cluster at the top of results, and this is where MMR becomes very useful. fetch_k is the size of the candidate pool that MMR selects from — starting with 10–20 is a common approach.

10.4.3) When to Stop Tuning

There are multiple parameters to adjust: k, fetch_k, metadata filters, distance thresholds, and more. Following these simple rules helps you tune efficiently:

Prepare several questions — some with relevant content in the documents and some without.
Run the questions and directly examine the retrieved chunks.
If a problem is found, change only one parameter at a time and re-test with the same questions.

If questions with relevant content produce correct answers and questions without relevant content result in abstention, you've achieved baseline quality. Getting the settings perfect from the start isn't easy. Respond to issues discovered during actual use and improve incrementally.