2. Agent Building Essentials

In Chapter 1, you made your first LLM call and saw the basic request-response flow. Now we need to understand what we're actually building: agentic AI systems. This chapter establishes the foundational concepts you'll use throughout the book.

By the end of this chapter, you'll understand:

• What makes an AI system "agentic" (and why it matters)
• Why frameworks like LangChain and LangGraph exist
• How LLMs actually work under the hood (and why this affects agent design)
• The economics of LLM usage (tokens, costs, and model selection)
• How to write effective prompts for agent systems

This is a conceptual chapter - we'll return to hands-on coding in Chapter 3. But these concepts are critical for understanding the design decisions you'll make when building agents.

2.1) What is Agentic AI? (Chatbot vs Agent)

When most people think of "AI applications," they picture chatbots: you ask a question, the AI responds, conversation ends. But agentic AI is fundamentally different. An agent doesn't just respond - it acts autonomously to achieve goals.

The Core Distinction

Let's start with a clear definition:

Chatbot: A system that responds to user input with generated text. The interaction is purely conversational. The chatbot has no ability to take actions in the world beyond generating its next message.

Agent: A system that can:

1. Perceive its environment (e.g., read order status, check inventory)
2. Decide what actions to take (e.g., choose between: create ticket, send email, or escalate)
3. Act on those decisions (e.g., call API, update database)
4. Iterate until it achieves a goal (e.g., verify order → create ticket → confirm → done)

The key difference is autonomy and action. A chatbot is reactive; an agent is proactive.

Concrete Example: Customer Support

Let's see this distinction in practice.

Chatbot Approach:

User: "I want a refund for order #12345"
 
Chatbot: "I understand you'd like a refund. To process this, please:
1. Visit our refund portal at example.com/refunds
2. Enter your order number
3. Select a reason
4. Submit the form
5. Wait 24-48 hours for manual review
 
Is there anything else I can help you with?"

The chatbot provides information but takes no action. The user must do all the work.

Agent Approach:

User: "I want a refund for order #12345"
 
Agent (internal reasoning):
1. User wants refund for order #12345
2. I need to verify this order exists
3. [Calls get_order_details(order_id="12345")]
4. Order found, eligible for refund
5. [Calls create_refund_ticket(order_id="12345", reason="customer_request")]
6. Ticket created: TICKET-789
 
Agent: "I've created refund ticket TICKET-789 for order #12345. 
Our refund team will process this within 3-5 business days. 
You'll receive an email confirmation shortly."

The agent took action: it verified the order, created a ticket, and confirmed the outcome. The user's problem is solved without manual steps.

Why This Matters for Development

Understanding this distinction shapes how you design your system:

Chatbot Development:

• Focus on response quality and conversation flow
• Primary concern: generating helpful, accurate text
• Simple architecture: prompt → LLM → response
• No need for external integrations

Agent Development:

• Focus on decision-making and action execution
• Primary concerns: choosing correct actions, handling errors, maintaining state
• Complex architecture: perception → reasoning → action selection → execution → verification
• Requires tool integrations, error handling, state management

The Spectrum of Autonomy

Not all agents are equally autonomous. There's a spectrum:

Level 1: Assisted Actions

• Agent suggests actions, user approves each one
• Example: "I can create a refund ticket. Should I proceed?"
• Safest approach for high-stakes operations

Level 2: Bounded Autonomy

• Agent acts within predefined constraints
• Example: Can create tickets and send emails, but cannot process refunds over $500 or access payment systems directly
• Most common in production systems (balance of efficiency and safety)

Level 3: Full Autonomy

• Agent acts independently to achieve goals
• Example: Handles entire refund workflow without human intervention
• Requires robust guardrails and monitoring

Key Characteristics of Agents

To summarize, an agentic AI system has these core properties:

1. Tool-Using: Can call functions, APIs, and external services (Without this, it's just a chatbot)
2. Goal-Directed: Works toward specific outcomes, not just responding
3. Multi-Step: Breaks complex tasks into sequences of actions
4. Adaptive: Adjusts behavior based on intermediate results
5. Stateful: Maintains context across multiple interactions

The first two are essential - without tools and goals, you don't have an agent. The rest are quality factors that separate good agents from great ones.

Now you understand what agents are and why they're powerful.

2.2) Why LangChain and LangGraph?

You might be wondering: "Why do I need frameworks? Can't I just call the OpenAI API directly?" Let's explore why frameworks exist and what problems they solve.

The Complexity of Agent Development

Building a simple chatbot with raw API calls is straightforward:

import openai
 
response = openai.chat.completions.create(
    model="gpt-5-mini",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

This works fine for basic use cases. But as soon as you want to build an agent, complexity explodes:

Challenge 1: Multi-Step Workflows Your agent needs to:

• Retrieve relevant documents from a knowledge base
• Decide which tool to call based on user intent
• Execute the tool and handle errors
• Format results and respond to the user

Each step requires careful orchestration, error handling, and state management.

Challenge 2: Provider Abstraction What if you want to:

• Switch from OpenAI to Anthropic or Google?
• Use different models for different tasks?
• Fall back to a cheaper model if the primary one fails?

With raw API calls, you'd need to rewrite significant code for each provider.

Challenge 3: Conversation Memory Agents need to remember context:

• Previous messages in the conversation
• Retrieved documents from earlier queries
• Intermediate results from tool calls

Managing this state manually is error-prone and tedious.

Challenge 4: Tool Integration Your agent needs to:

• Define available tools with schemas
• Let the LLM choose which tool to call
• Parse tool arguments from LLM output
• Execute tools safely with validation
• Handle tool errors and retry logic

This requires significant boilerplate code and security considerations at every step.

Challenge 5: Complex Routing Real agents need conditional logic:

• "If user asks about refunds, retrieve policy docs"
• "If user wants a refund, create a ticket"
• "If question is off-topic, politely decline"

Implementing this with if-else statements becomes unmaintainable quickly.

What LangChain Provides

LangChain is a framework for building LLM applications. It provides:

1. Model Abstraction

Unified interface for different LLM providers (OpenAI, Anthropic, Google, etc.).

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic
 
# Same interface, different providers
openai_llm = ChatOpenAI(model="gpt-5-mini")
anthropic_llm = ChatAnthropic(model="claude-4-5-sonnet")
 
# Both use .invoke() with same message format
response = openai_llm.invoke([{"role": "user", "content": "Hello"}])

Switch providers without rewriting your application logic.

2. Composable Chains (LCEL)

LangChain Expression Language - a syntax for connecting components into pipelines.

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
 
# Compose a pipeline with | operator (like Unix pipes)
chain = prompt | llm | output_parser
 
# Execute the entire pipeline with one call
result = chain.invoke({"input": "user question"})

Build complex workflows without manual data passing between steps. We'll learn LCEL in Chapter 6.

3. Conversation Memory

Message history management for stateful conversations.

from langchain_core.chat_history import InMemoryChatMessageHistory
 
# Message history helper
history = InMemoryChatMessageHistory()
history.add_user_message("Hi")
history.add_ai_message("Hello!")
 
# Retrieve messages when needed
messages = history.messages
response = llm.invoke(messages)

Abstract message storage with helper classes instead of managing lists manually. At this stage, history is still passed explicitly to the model. We'll integrate this into chains in Chapter 8. LangGraph (Chapters 15+) makes this even simpler with built-in state management.

4. Tool Integration

Decorator-based system for exposing Python functions to LLMs.

from langchain_core.tools import tool
 
@tool
def create_ticket(order_id: str, reason: str) -> str:
    """Create a support ticket for an order."""
    # Implementation here
    return f"Created ticket for {order_id}"
 
# LangChain handles schema generation and LLM integration

Turn any Python function into a tool that can be discovered and called by tool-enabled LLMs or agents — no manual JSON schema writing.

5. Document Loaders and Vector Stores

Ready-to-use components for loading documents from various sources and storing them as searchable embeddings.

from langchain_community.document_loaders import TextLoader
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
 
# Load documents
docs = TextLoader("support_docs.txt").load()
 
# Create searchable index
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(docs, embeddings)
 
# Retrieve relevant docs
results = vectorstore.similarity_search("refund policy")

Build RAG systems using prebuilt loaders and vector stores instead of implementing parsing, embedding, and retrieval from scratch.

What LangGraph Provides

LangGraph extends LangChain for complex agent workflows. It provides:

1. Explicit State Management

Define all agent data in a single typed schema instead of scattering it across variables.

from typing import TypedDict, Optional
from langchain_core.messages import BaseMessage
from langchain_core.documents import Document
 
class AgentState(TypedDict):
    messages: list[BaseMessage]
    retrieved_docs: list[Document]
    current_step: Optional[str]
 
# All agent data lives here - one place to inspect when debugging

No more hunting for where data lives or how it flows between steps. State changes are explicit: nodes read from state and return updates. Your IDE autocompletes field names, and type checkers catch errors before runtime.

2. Graph-Based Workflows

Build workflows by declaring steps (nodes) and their connections (edges), not by writing orchestration code.

graph = StateGraph(AgentState)
 
# Define nodes (steps in your workflow)
graph.add_node("retrieve", retrieve_docs)
graph.add_node("answer", generate_answer)
 
# Define edges (transitions between steps)
graph.add_edge("retrieve", "answer")

You define the structure—"retrieve runs, then answer runs"—and LangGraph handles execution. There is no need to write orchestration code to pass state between steps. Control flow such as sequencing or branching is declared in the graph structure itself.

3. Conditional Routing

Runtime decision-making about which path to take in the workflow.

from langchain_core.messages import BaseMessage
 
def route_request(state):
    last_message = state["messages"][-1].content
    if "refund" in last_message:
        return "create_ticket"
    else:
        return "answer_question"
 
graph.add_conditional_edges(
    "classify",
    route_request,
    {
        "create_ticket": "create_ticket",
        "answer_question": "answer_question",
    }
)

The key insight: The routing function returns the name of the next node ("create_ticket" or "answer_question").

Why this matters: Your decision logic is separate from execution. Change the routing rules? Edit one function. See all possible paths? Look at the graph definition. Debug which path was taken? Inspect the execution trace—no digging through nested function calls.

4. Checkpointing and Persistence

State is automatically checkpointed after each step, enabling crash recovery and pause-resume workflows.

from langgraph.checkpoint.sqlite import SqliteSaver
 
checkpointer = SqliteSaver("agent_state.db")
graph = graph.compile(checkpointer=checkpointer)
 
# State is automatically saved at each step
result = graph.invoke(
    input_state,
    config={"configurable": {"thread_id": "user-123"}}
)

After each step, LangGraph checkpoints the current state to persistent storage. If the process crashes, execution can resume from the latest checkpoint associated with the same thread ID.

Checkpointing enables pause-and-resume workflows, such as waiting for human approval, when combined with conditional routing or interrupts.

When to Use Each Framework

Use LangChain when:

• Building simple chains (prompt → LLM → parser)
• Implementing RAG systems
• Handling message-based context (passing chat history explicitly)
• Abstracting across LLM providers

Use LangGraph when:

• Building multi-step agent workflows
• Implementing conditional routing logic
• Managing complex state across steps
• Requiring checkpointing and crash recovery

2.3) How LLMs Work (for Agent Builders)

To build effective agents, you need to understand how LLMs actually work. This isn't about the mathematics of transformers - it's about the mental model that shapes how you design agent systems.

The Core Mechanism: Token Prediction

Here's the fundamental insight: LLMs don't "know" facts in the way databases do. They predict the next token.

Let's break this down with a concrete example.

Input: "The capital of France is"

What you might think happens:

1. LLM "looks up" the capital of France in its knowledge base
2. LLM "retrieves" the answer: Paris
3. LLM returns "Paris"

What actually happens:

1. LLM converts input to tokens: ["The", "capital", "of", "France", "is"]
2. LLM calculates probability distribution over ALL possible next tokens
3. Most likely next token: "Paris" (highest probability)
4. LLM samples from distribution (usually picking highest probability)
5. LLM returns "Paris"

The LLM doesn't "know" Paris is the capital. It predicts "Paris" is the most likely next token given the input pattern.

(Note: Tokenization and probabilities are conceptual and vary by model and tokenizer.)

Why This Matters for Agents

This token prediction model has profound implications for agent design:

Implication 1: LLMs Can Hallucinate

Because LLMs predict tokens (not retrieve facts), they can generate plausible-sounding but incorrect information.

from langchain_openai import ChatOpenAI
 
llm = ChatOpenAI(model="gpt-4o-mini")
response = llm.invoke("What is the capital of Atlantis?")
print(response.content)
 
# Note: Actual output may vary. 
# Modern models might recognize Atlantis as fictional and refuse to answer.
# The key point:
# without explicit fact-checking, LLMs can generate plausible-sounding incorrect information.

Possible (historical or unconstrained) output:

The capital of Atlantis is Etheria. According to Plato's records, Etheria was a port city located on the easternmost tip of the island, and its name was derived from the belief that it reached up to the heavens.

The LLM generates a reasonable-sounding answer even though Atlantis is fictional. For agents, this means:

• Never trust LLM output blindly
• Validate facts against authoritative sources
• Implement guardrails

Implication 2: Context is Everything

LLMs only see the tokens you provide. They have no memory of previous conversations unless you explicitly include that context.

# First call
response1 = llm.invoke("My name is Alice")
print(response1.content)  # "Nice to meet you, Alice!"
 
# Second call (separate invocation)
response2 = llm.invoke("What's my name?")
print(response2.content)  # "I don't have access to your name..."

The second call has no context from the first. For agents, this means:

• You must manage conversation history
• Context window limits matter
• State management is critical

Implication 3: Prompts Are Instructions, Not Queries

Because LLMs predict tokens, the way you phrase your prompt dramatically affects output quality.

# Weak prompt (query-style)
response = llm.invoke("refund policy")
# Output: "What would you like to know about the refund policy?"
 
# Strong prompt (instruction-style)
response = llm.invoke(
    "You are a customer support agent. Explain our refund policy clearly and concisely."
)
# Output: "Our refund policy allows returns within 30 days..."

For agents, this means:

• Prompts are your primary control mechanism
• Prompt engineering is a core skill
• System messages set agent behavior

Implication 4: Structured Output Requires Guidance

LLMs naturally generate free-form text. Getting structured output (JSON, specific formats) requires explicit instruction.

# Without structure guidance
response = llm.invoke("Extract the order ID from: 'I want a refund for order #12345'")
print(response.content)
# Output: "The order ID is 12345" (plain text, inconsistent format)
 
# With structure guidance
response = llm.invoke(
    'Extract the order ID and return ONLY a JSON object with format: {"order_id": "..."}\n\n'
    "Text: 'I want a refund for order #12345'"
)
print(response.content)
# Output: {"order_id": "12345"} (structured, parseable)

For agents, this means:

• Use schemas to constrain output format
• Explicitly specify output formats
• Validate and parse LLM responses

Free-form text is optimized for humans. Agents require explicit formatting to ensure machine-readability.

Determinism, Stochasticity, and Modern LLMs

LLMs are fundamentally probabilistic systems. They generate text by predicting the most likely next tokens, not by executing deterministic rules. As a result, the same input does not always guarantee the same output.

In earlier models, developers explicitly controlled this randomness using parameters such as temperature. Lower values produced more predictable outputs, while higher values encouraged variation and creativity.

Many modern reasoning-oriented models no longer expose parameters like temperature or top_p. Instead, they manage decoding and sampling strategies internally to prioritize stable, structured reasoning. However, this does not mean these models are fully deterministic.

Even these models don't guarantee identical outputs. Outputs may vary due to:

• Internal sampling: The model may follow different reasoning paths, producing outputs that vary in structure, detail, or phrasing.
• Model updates: Providers continuously update models without notice, so the same prompt can yield different responses over time.
• Safety filters: Content moderation may cause the model to answer directly in one case but hedge, refuse, or rephrase in another.
• Tool policies: In agent systems, the model may invoke different tools—or no tools—for the same input, changing execution paths.

What this means for agent builders:

The key concern is not parameter tuning—it's predictability. Agent systems should be designed assuming that LLM outputs can vary in wording, structure, or even conclusions unless explicitly constrained.

This leads to several concrete design principles for agent systems:

• Never rely on exact wording for logic decisions — control flow should depend on structured signals (schemas, enums, flags), not on matching specific phrases in model output.
• Force structure at boundaries — whenever an LLM output is consumed by code, constrain it using schemas, validators, or strict formats so the program never has to "interpret" free text.
• Verify anything that matters — facts that affect money, permissions, or irreversible actions must be checked with tools or external systems, not trusted from the model alone.
• Treat LLM outputs as proposals, not decisions — the model suggests what to do, but the system decides whether, when, and how to act.

In modern agent systems, reliability comes from system design, not from parameter tuning. The more critical the task, the less freedom the model should have—and the more structure your agent should enforce.

Key Takeaway: Build reliable agents through schemas, validation, and tool integration—not by hoping for consistent LLM outputs.

In the next section, we'll explore the economic implications of token-based processing.

2.4) Tokens: The Fundamental Resource

Tokens are the basic unit LLMs process. Understanding tokens is essential because they define both what's possible (constraints) and what's expensive (costs) in agent systems.

What Is a Token?

A token is the smallest unit of text an LLM reasons over and generates.

Depending on the language and context, a token may represent:

• A word (agent)
• Part of a word (calculat, ion)
• A number or symbol (#, 123)
• Punctuation or whitespace

Tokens are not characters and not words — they are model-specific units created by the tokenizer.

Every piece of information sent to or generated by the model is measured in tokens:

• System instructions
• User messages
• Retrieved documents
• Tool descriptions
• Model outputs

Tokens are the fundamental currency of LLM interaction.

Tokens as a System Constraint

Tokens aren't just a cost—they're a hard limit on what your agent can do in a single request.

Every LLM has a context window: a fixed maximum number of tokens it can process at once.

Example scenario: Your customer support agent needs:

• System instructions: 200 tokens
• Last 10 messages: ~2,000 tokens
• 3 retrieved help articles: ~1,500 tokens
• Generated response: ~200 tokens
• Total: 3,900 tokens

If your model's context window is 4,000 tokens, you're at 97.5% capacity. One more long message and the system stops working.

(Modern models typically have 128K+ token context windows, but the principle remains: context is finite, and you must design around this limit.)

What happens when you exceed the limit:

• Older messages get dropped → Agent forgets earlier context, breaking conversation continuity
• Retrieved documents get truncated → Critical information is lost, leading to incorrect answers
• The request fails entirely → System cannot respond at all

You can't pay for more space. The context window is a hard limit—like trying to fit 2 liters in a 1-liter bottle.

This is why long-running agents must actively manage what stays in context and what does not. Token management is a core architectural concern, not an optimization detail.

What Tokens Directly Affect

Beyond the immediate context window limit, tokens shape two critical design decisions:

1. Memory Strategy: Full History vs Summarization

Keeping the full conversation history preserves detail but causes token usage to grow continuously.

A common alternative is memory summarization:

• Replace older messages with a compact summary
• Preserve intent while reducing token cost

This trade-off affects:

• Cost
• Accuracy
• Long-term agent consistency

Memory design is therefore a token management problem.

2. RAG Chunk Size and Retrieval Strategy

In Retrieval-Augmented Generation (RAG), documents are split into chunks before retrieval.

• Large chunks

  • Fewer retrieval calls
  • Higher token cost per request
  • More irrelevant context

• Small chunks

  • Lower token cost
  • Higher precision
  • Risk of missing key information

Chunk size is a critical design decision — it directly impacts both cost and answer quality.

Token Economics

Understanding token costs helps you build cost-effective systems.

Typical Pricing (2026):

Key Insight: Output tokens cost 4–8x more than input tokens, which means uncontrolled generation is often the largest cost driver in production systems.

Quick Cost Estimate:

For a typical request with 500 input tokens and 50 output tokens using GPT-5-mini:

• Input: (500 / 1,000,000) × $0.25 = $0.000125
• Output: (50 / 1,000,000) × $2.00 = $0.0001
• Total: ~$0.000225 per request

At 10,000 requests/day: ~$67.5/month

Cost Optimization in Practice

Strategy 1: Match Model to Task Complexity

Use smaller, cheaper models for simple tasks:

from langchain_openai import ChatOpenAI
 
# Expensive model for complex reasoning
complex_llm = ChatOpenAI(model="gpt-5")
 
# Cheap model for simple tasks
simple_llm = ChatOpenAI(model="gpt-5-nano")
 
def get_llm_for_task(task_type):
    if task_type == "complex_reasoning":
        return complex_llm
    else:
        return simple_llm

Strategy 2: Balance Context Size with Quality

Include only necessary context:

# Efficient: Include only relevant chunks
relevant_chunks = retrieve_top_k(user_question, k=3)  # ~500 tokens
prompt = f"Context: {relevant_chunks}\n\nQuestion: {user_question}"

Important: Overly aggressive context reduction can hurt answer accuracy. Balance cost savings with quality.

Strategy 3: Control Output Length

Limit how much the model generates:

# Cost-controlled
llm = ChatOpenAI(model="gpt-5-mini", max_tokens=100)
response = llm.invoke(messages)
# Maximum 100 output tokens

Key Takeaway: Token management is not just about reducing costs—it's about designing reliable, scalable agent systems within hard resource constraints.

In the next section, we'll explore the model landscape and learn how to choose the right model for each task.

2.5) Understanding the Model Landscape

Choosing the right LLM for your agent requires balancing:

• Context window: How much text can the model process?
• Cost: How much does each request cost?
• Latency: How fast does the model respond?
• Capability: How well does the model reason?

Let's survey the 2026 model landscape.

Major Model Families

OpenAI GPT Models

Anthropic Claude Models

Google Gemini Models

Provider-Specific Features

OpenAI:

• Best for: General-purpose agents, broad multi-domain workflows, function calling
• Strong at: Versatile reasoning, strong developer tooling & ecosystem, frequent model updates

Anthropic:

• Best for: Safety-sensitive workflows, structured and extended reasoning
• Strong at: Deep analysis, methodical outputs, extended thinking with strong alignment

Google:

• Best for: Massive context ingestion and multimodal tasks
• Strong at: Large-scale document analysis, multimodal understanding, high-throughput processing

Matching Models to Tasks

Choose based on task complexity, context size, and latency needs:

By Task Complexity

Simple Tasks → Cost-optimized models (GPT-5-nano, GPT-5-mini):

• Intent classification, sentiment analysis, keyword extraction, simple formatting
• Choose when: Cost is the primary concern

Moderate Tasks → Balanced model (GPT-5-mini):

• Question answering, summarization, tool selection
• Choose when: Need to balance cost and quality

Complex Tasks → Performance-focused models (GPT-5, Claude Sonnet, Gemini Pro):

• Multi-step reasoning, code generation, detailed analysis
• Choose when: Reasoning capability matters, cost is acceptable

Highest Complexity → Premium models (Claude Opus):

• Deeply complex reasoning, mission-critical decisions
• Choose when: Accuracy is paramount, cost is secondary

By Latency Needs

Real-Time (~1s or less perceived latency) → Fast models (GPT-5-nano, Gemini Flash):

• User-facing chat
• Interactive applications

Near Real-Time (1-3s) → Most models:

• Standard agent tasks

Batch (>3s) → Capable models:

• Background analysis

Context Window Considerations

Standard tasks: All major models support 200K+ tokens, sufficient for most agent workflows.

Special cases:

• Need 400K tokens: GPT-5 family (full document analysis, large codebases)
• Need 1M tokens: Gemini models (entire books, massive document sets)

Practical advice: Even with large context windows, selective retrieval (RAG) usually produces better results.

Key Takeaways

1. No single "best" model → Different models excel at different tasks
2. Match model to task complexity → Don't overpay for simple tasks
3. Context window ≠ better → Use selective retrieval
4. Latency affects user experience → Consider response time for interactive tasks

In practice: Most agents use multiple models—cheap models for simple tasks, capable models for complex reasoning. We'll implement this in later chapters.

In the next section, we'll learn how to control model behavior through effective prompting.

2.6) Prompting Basics for Agent Systems

Prompts are your primary interface for controlling LLM behavior. For agents, effective prompting is critical - it determines whether your agent makes correct decisions, calls the right tools, and produces reliable outputs.

The Anatomy of a Prompt

A complete prompt has three components:

1. System Message (Role and Constraints) Defines the agent's persona, capabilities, and boundaries.

2. Context (Relevant Information) Provides the information needed to complete the task.

3. Instruction (Specific Task) Tells the agent exactly what to do.

Let's see this in practice:

from langchain_openai import ChatOpenAI
 
llm = ChatOpenAI(model="gpt-5-mini")
 
response = llm.invoke([
    # System message: Define role and constraints
    {
        "role": "system",
        "content": """You are a customer support agent for TechCorp.
        
Your capabilities:
- Answer questions about refund policies
- Create support tickets
- Provide troubleshooting guidance
 
Your constraints:
- Only answer questions about TechCorp products
- Never make promises about refund timelines
- Always be polite and professional"""
    },
    
    # User message: Context + Instruction
    {
        "role": "user",
        "content": """Context: Customer purchased laptop model X500 on 2026-01-15.
Today is 2026-02-20. Our refund policy allows returns within 30 days.
 
Instruction: The customer wants a refund. What should I tell them?"""
    }
])
 
print(response.content)

Output:

I understand you'd like a refund for your laptop model X500. Unfortunately, 
since your purchase was on January 15th and today is February 20th, we're 
beyond our 30-day return window. However, I'd be happy to create a support 
ticket to explore other options, such as warranty service or exchange. 
Would you like me to proceed with that?

System Messages: Setting Agent Behavior

The system message is where you define your agent's personality and capabilities. This is the most important part of agent prompting.

Weak System Message:

system_message = "You are a helpful assistant."

Strong System Message:

system_message = """You are a customer support agent for TechCorp.
 
ROLE:
You help customers with refund requests, product questions, and technical issues.
 
CAPABILITIES:
- Answer questions using provided documentation
- Create support tickets when needed
- Provide step-by-step troubleshooting
 
CONSTRAINTS:
- Only answer questions about TechCorp products
- If you don't know something, say so - never guess
- Always cite sources when using documentation
- Never promise specific timelines or outcomes
 
TONE:
Professional, empathetic, and solution-focused.
"""

Why the strong version works better:

1. Explicit capabilities → Agent knows what it can do
2. Clear constraints → Agent knows what to avoid
3. Defined tone → Consistent personality
4. Specific instructions → Reduces ambiguity

Instruction Clarity: Be Specific

LLMs follow instructions literally. Vague instructions produce unreliable results.

Vague Instruction:

instruction = "Help the customer with their refund."

Specific Instruction:

instruction = """Analyze the customer's request and determine:
1. Is the order eligible for refund? (Check purchase date vs refund policy)
2. If eligible: Explain the refund process
3. If not eligible: Explain why and offer alternatives
 
Format your response as:
- Eligibility: [YES/NO]
- Reason: [Brief explanation]
- Next Steps: [What the customer should do]
"""

Context Management: Provide What's Needed

Agents need context to make decisions, but too much context wastes tokens and confuses the model.

Over-Contextualization (Wasteful):

context = f"""
Company History: TechCorp was founded in 1995...
Product Catalog: We sell 500+ products including...
Refund Policy: {refund_policy_text}
Shipping Policy: {shipping_policy_text}
Warranty Policy: {warranty_policy_text}
Customer History: {full_customer_history}
"""
# 5000+ tokens, most irrelevant

Selective Context (Efficient):

context = f"""
Relevant Policy: {refund_policy_text}
Order Details: {order_details}
"""
# 200 tokens, all relevant

Output Formatting: Structure Your Responses

For agents, you often need structured output (JSON, specific formats) rather than free-form text.

Unstructured Output (Hard to Parse):

response = llm.invoke([
    {"role": "system", "content": "You are a support agent."},
    {"role": "user", "content": "Should we create a ticket for this refund request?"}
])
print(response.content)
# Output: "Yes, I think we should create a ticket because..."
# Problem: Hard to parse, inconsistent format

Structured Output (Easy to Parse):

response = llm.invoke([
    {"role": "system", "content": """You are a support agent.
    
Always respond in this JSON format:
{
  "action": "CREATE_TICKET" or "ANSWER_QUESTION" or "ESCALATE",
  "reason": "Brief explanation",
  "response_text": "What to tell the customer"
}"""},
    {"role": "user", "content": "Customer wants refund for order #12345, purchased 40 days ago"}
])
print(response.content)

Output:

{
  "action": "CREATE_TICKET",
  "reason": "Order is outside 30-day refund window, needs manual review",
  "response_text": "I've created a support ticket to review your refund request. Our team will contact you within 24 hours."
}

We'll use Pydantic schemas for robust structured output in Chapter 7.

Few-Shot Examples: Show, Don't Just Tell

For complex tasks, providing examples is more effective than lengthy instructions.

Zero-Shot (Instructions Only):

prompt = """Extract the order ID, product name, and issue from customer messages.
 
Customer message: "My laptop X500 order #12345 won't turn on"
"""
# Model might struggle with format

Few-Shot (With Examples):

prompt = """Extract the order ID, product name, and issue from customer messages.
 
Example 1:
Input: "My laptop X500 order #12345 won't turn on"
Output: {"order_id": "12345", "product": "laptop X500", "issue": "won't turn on"}
 
Example 2:
Input: "Order 67890 - phone not charging"
Output: {"order_id": "67890", "product": "phone", "issue": "not charging"}
 
Now extract from this message:
Input: "My tablet order #11111 has a cracked screen"
Output:
"""

The model learns the pattern from examples and applies it consistently.

Prompt Engineering Patterns for Agents

Pattern 1: Chain of Thought (Reasoning)

For complex decisions, ask the model to "think step by step":

prompt = """You need to decide whether to create a support ticket.
 
Think through this step by step:
1. What is the customer asking for?
2. Can this be answered with existing documentation?
3. Does this require manual intervention?
4. What action should we take?
 
Customer message: "I want a refund for order #12345 but I lost the receipt"
 
Reasoning:
"""

The model will explicitly show its reasoning, making decisions more transparent and reliable.

Pattern 2: Constrained Generation (Safety)

Limit the model's possible outputs:

prompt = """Classify the customer's intent. Respond with EXACTLY ONE of these options:
- REFUND_REQUEST
- PRODUCT_QUESTION
- TECHNICAL_ISSUE
- OFF_TOPIC
 
Customer message: "How do I reset my password?"
 
Classification:
"""

This prevents the model from generating unexpected outputs. By explicitly constraining output options:

• Prevents parsing errors (always one of four defined options)
• Blocks unintended actions (prevents execution of undefined actions)
• Simplifies debugging (limited output space makes issues easier to trace)

Pattern 3: Self-Critique (Quality)

Ask the model to verify and improve its own output:

prompt = """Generate a response to the customer, then critique it.
 
Customer message: "I want a refund"
 
Step 1 - Generate response:
[Your response here]
 
Step 2 - Critique:
- Is this response accurate?
- Is it helpful?
- Does it follow company policy?
- What could be improved?
 
Step 3 - Final response (incorporating critique):
[Improved response here]
"""

This multi-step approach often produces higher-quality outputs because:

• Catches errors early: The model reviews its own reasoning before finalizing
• Improves tone and clarity: Self-reflection helps identify unclear or inappropriate language
• Ensures policy compliance: The critique step verifies adherence to guidelines

Common Prompting Mistakes

Mistake 1: Assuming the Model "Knows" Things

LLMs don't have access to real-time information or implicit context. Always provide all necessary data explicitly.

# Bad: Assumes model knows current date
prompt = "Is this order eligible for refund? Order #12345"
 
# Good: Provide all necessary information
prompt = f"""Is this order eligible for refund?
Order #12345
purchased {purchase_date}
Today: {current_date}
Policy: 30-day returns
"""

Mistake 2: Ambiguous Instructions

Vague verbs like "handle," "process," or "deal with" leave too much room for interpretation. Be explicit about the exact action needed.

# Bad: What does "handle" mean?
prompt = "Handle this refund request"
 
# Good: Explicit action
prompt = "Determine if this refund request is eligible. If yes, create a ticket. If no, explain why."

Mistake 3: Overloading with Context

Including irrelevant information wastes tokens, increases latency, and can confuse the model. Retrieve only what's needed for the specific task.

# Bad: 10,000 tokens of context, most irrelevant
prompt = f"""
{entire_knowledge_base}
 
Question: What's the refund policy?
"""
 
# Good: Retrieve only relevant section
prompt = f"""
{refund_policy_section}
 
Question: What's the refund policy?
"""

Mistake 4: Inconsistent Formatting

When output format varies, downstream code breaks. Always specify the exact format you expect, especially for structured data.

# Bad: Sometimes JSON, sometimes plain text
prompt = "Respond with your decision"
 
# Good: Always specify format
prompt = 'Respond in JSON format: {"decision": "...", "reason": "..."}'

Key Takeaways

Effective prompting for agents requires:

1. Clear system messages → Define role, capabilities, constraints
2. Specific instructions → Tell the model exactly what to do
3. Selective context → Provide only relevant information
4. Structured output → Specify format explicitly
5. Few-shot examples → Show the pattern you want
6. Reasoning prompts → Ask for step-by-step thinking

Prompting is a skill that improves with practice. Throughout this book, you'll see these patterns applied in real agent systems.

Chapter Summary

You now understand the foundational concepts for building agentic AI systems:

Agentic AI vs Chatbots:

• Agents act autonomously to achieve goals
• Agents use tools and make multi-step decisions
• Agents maintain state and adapt based on results

Why Frameworks Matter:

• LangChain provides model abstraction, chains, memory, tools, and RAG
• LangGraph adds state management, routing, and checkpointing
• Frameworks reduce boilerplate and enable complex workflows

LLM Mechanics:

• LLMs predict tokens, they don't retrieve facts
• Context is explicit, not implicit
• Prompts are instructions, not queries
• Structured output requires guidance
• Context windows are finite

Token Economics:

• Output tokens cost 4–8x more than input tokens
• Model choice has 10-100x cost impact
• Selective context reduces costs dramatically
• Monitoring and budgets prevent runaway spending

Model Selection:

• Match model to task complexity
• Consider context requirements and latency constraints
• Use multi-model architectures for cost optimization
• Start cheap, upgrade only if needed

Prompting Fundamentals:

• System messages define agent behavior
• Specific instructions produce reliable results
• Selective context improves quality and reduces cost
• Structured output enables parsing and validation
• Few-shot examples teach patterns effectively