3. Build Your First Streaming CLI Chat

In Chapter 1, you made your first LLM call and saw a complete response appear at once. In Chapter 2, you learned the conceptual foundations of agentic AI and why LangChain exists. Now it's time to build something practical: a streaming chat application that feels responsive and professional.

Why streaming matters: When you ask an LLM a complex question, waiting 10-30 seconds for a complete response feels broken. Streaming lets tokens appear as they're generated, creating a natural conversational flow. This chapter builds a CLI chat application with streaming output, proper configuration management, debugging capabilities, and robust error handling.

What you'll build: By the end of this chapter, you'll have a working chat.py script that:

• Streams LLM responses token-by-token to the terminal
• Loads API keys securely from environment variables
• Handles different model types (chat vs reasoning models) with appropriate parameters
• Provides debugging tools to inspect what's actually sent to the LLM
• Gracefully handles common errors (missing API keys, network failures, invalid inputs)

3.1) Create a Working Folder and Install Packages

Before writing any code, you need a clean project structure and the right dependencies. This section establishes the foundation for a maintainable Python project.

Project Structure

Create a new directory for your chat application:

mkdir langchain-chat
cd langchain-chat

Python Environment Setup

Create a virtual environment to isolate dependencies:

# Create virtual environment
python -m venv venv
 
# Activate it (macOS/Linux)
source venv/bin/activate
 
# Activate it (Windows)
venv\Scripts\activate

Why virtual environments? LangChain has many dependencies (e.g., OpenAI SDK, Pydantic, async libraries). A virtual environment ensures:

• Your system Python stays clean
• Different projects can use different LangChain versions
• Dependencies are reproducible (via requirements.txt)

You'll see (venv) in your terminal prompt when activated.

Installing LangChain

Install the core LangChain packages:

pip install langchain-core==1.2.7 langchain-openai==1.1.7 python-dotenv

Package breakdown:

• langchain-core: Core abstractions (messages, prompts, chains, runnables)
• langchain-openai: OpenAI-specific implementations (ChatOpenAI, embeddings)
• python-dotenv: Loads environment variables from .env files

Version note: This book uses LangChain 1.2.x as of January 2026. If you're reading this in the future, check the LangChain documentation for the latest version.

Verify Installation

Create a simple test to confirm everything works:

# test_install.py
try:
    from langchain_core.messages import HumanMessage
    from langchain_openai import ChatOpenAI
    print("✓ langchain-core: OK")
    print("✓ langchain-openai: OK")
    print("\nInstallation successful!")
except ImportError as e:
    print(f"✗ Import failed: {e}")
    print("Ensure your virtual environment is activated.")

Run it:

python test_install.py

Expected output:

✓ langchain-core: OK
✓ langchain-openai: OK
 
Installation successful!

If you see "Installation successful!", you're ready to proceed. If you get an import error, double-check that:

• Your virtual environment is activated (look for (venv) in your prompt)
• The packages were installed successfully (try running pip list)

Creating requirements.txt

You just installed packages with pip install commands. While this works for learning, there's a better way: requirements.txt files. This is standard practice in Python projects for several reasons:

Why use requirements.txt?

• Reproducibility: Others (or you in 6 months) can install the exact same package versions
• Clear dependency management: See at a glance which packages your project needs
• Team collaboration: Team members use identical versions, avoiding "works on my machine" issues
• Automation: Servers or CI/CD pipelines can set up the environment with one line: pip install -r requirements.txt

Create a requirements.txt file in your project root:

# requirements.txt
langchain-core==1.2.7
langchain-openai==1.1.7
python-dotenv

Note the syntax:

• ==1.2.7 pins to an exact version (recommended for reproducibility)
• No version specifier (like python-dotenv) installs the latest stable version
• Lines starting with # are comments

Now anyone can install all dependencies with a single command:

pip install -r requirements.txt

This is much better than typing out each package individually. If a teammate clones your project, they just need to:

1. Create a virtual environment
2. Run pip install -r requirements.txt

No need to remember package names or versions—it's all in the file.

Your Project Structure

After completing this section, your folder should look like:

langchain-chat/
├── venv/                 # Virtual environment (don't commit to git)
├── requirements.txt      # Dependency list
└── test_install.py       # Installation verification script

Next: Section 3.2 shows how to securely load API keys using .env files.

3.2) Environment Variables with .env

API keys are secrets. Hardcoding them in your code is a security risk (especially if you commit to git). This section shows the standard approach: environment variables loaded from a .env file.

Why Environment Variables?

The problem with hardcoded keys:

# ❌ NEVER DO THIS
llm = ChatOpenAI(api_key="sk-proj-abc123...")

If you commit this code to GitHub, your API key is public. Anyone can use it, run up charges on your account, or get your key revoked.

The solution: Store secrets in environment variables, load them at runtime.

Creating the .env File

Create a .env file in your project root:

# .env
OPENAI_API_KEY=sk-proj-your-actual-key-here

Get your API key:

1. Go to platform.openai.com/api-keys
2. Create a new secret key
3. Copy it immediately (you can't view it again)
4. Paste it into your .env file, replacing sk-proj-your-actual-key-here

Critical security step: Before you do anything else, protect your API key from being committed to git.

Create a .gitignore file in your project root and add these lines:

# .gitignore
venv/
__pycache__/
*.pyc
.env

The .env line tells git to ignore your API key file. This prevents accidentally committing secrets to version control.

Your project structure now:

langchain-chat/
├── venv/
├── .env                  # Your API key (ignored by git)
├── .gitignore           # Contains: .env, venv/, etc.
├── requirements.txt
└── test_install.py

Loading Environment Variables

The python-dotenv package loads .env files into os.environ:

# chat.py
import os
from dotenv import load_dotenv
 
# Load .env file
load_dotenv()
 
# Access environment variables
api_key = os.environ.get("OPENAI_API_KEY")
 
if not api_key:
    raise ValueError("OPENAI_API_KEY not found in environment")
 
print(f"API key loaded: {api_key[:8]}...")  # Show first 8 chars only

How load_dotenv() works:

1. Searches for a .env file starting from where you run the script
2. Reads each line in the format KEY=value
3. Adds each variable to os.environ
4. If a variable is already set (e.g., by your hosting platform), it won't be overwritten—the existing value stays

Using the API Key with LangChain

LangChain's OpenAI implementations (ChatOpenAI, etc.) automatically look for OPENAI_API_KEY in os.environ:

from langchain_openai import ChatOpenAI
 
load_dotenv()
 
# This automatically uses os.environ["OPENAI_API_KEY"]
llm = ChatOpenAI(model="gpt-4o-mini")

LangChain's convention: When you create ChatOpenAI() without an api_key parameter, it automatically looks for OPENAI_API_KEY in the environment. This is a standard pattern across LangChain integrations.

Explicit API key (for testing or multiple keys):

llm = ChatOpenAI(
    model="gpt-4o-mini",
    api_key=os.environ.get("OPENAI_API_KEY")
)

This is useful when you have multiple API keys (development vs production) or want to be explicit about which key is used.

Environment Variables in Production

In production environments (cloud platforms, Docker containers), you don't use .env files. Instead, you configure environment variables through the platform's settings:

• Docker: Use the -e flag when running containers
• Cloud platforms: Set environment variables in configuration dashboards
• CI/CD: Use secrets management tools

The important part: your code doesn't change. os.environ.get("OPENAI_API_KEY") works the same way whether the variable comes from a .env file or a cloud platform. We'll cover deployment in detail in later chapters.

Verify Your Setup

To confirm everything is working, you can test the environment variable loading code shown earlier. If your .env file is properly configured, os.environ.get("OPENAI_API_KEY") will return your API key.

If os.environ.get("OPENAI_API_KEY") returns None, verify that:

1. You called load_dotenv() before accessing the environment variable
2. .env exists in the project root
3. OPENAI_API_KEY=sk-proj-... is correctly written in .env
4. You're running from the project root directory

Next: Section 3.3 implements the actual chat loop with streaming output.

3.3) Implementing the Chat Loop with Streaming Output

Now you'll build the core chat loop. This section introduces streaming - the key difference between a sluggish chatbot and a responsive one.

Understanding Streaming

Without streaming (Chapter 1 approach):

response = llm.invoke("Write a 500-word essay on AI")
print(response.content)  # Wait 20 seconds, then entire essay appears

With streaming:

for chunk in llm.stream("Write a 500-word essay on AI"):
    print(chunk.content, end="", flush=True)  # Tokens appear as generated

Why streaming matters:

• Immediate feedback: Instead of staring at a blank screen for 20 seconds, you see words appearing right away
• Natural conversation feel: Just like talking to a person - responses come progressively, not all at once
• Save time and money: If the LLM starts giving the wrong answer, you can stop it early instead of waiting for a complete (useless) response
• Better debugging: When building applications, you can spot issues (like formatting errors) as they happen, not after a long wait

What streaming actually is: Streaming is incremental delivery of the same response text. It doesn't expose hidden reasoning or internal model processes - it just shows you partial output as it becomes available from the API. Think of it like downloading a file: you see progress as chunks arrive, but the file content is the same whether you download it all at once or in pieces.

Note on chunk boundaries: Chunks are not guaranteed to align with words or sentences. The API sends tokens in small batches for efficiency, so a chunk might be "Hel", "lo! How", " can I", " help you", "?". This is normal and expected - don't try to parse meaning from individual chunks.

The Basic Chat Loop

Here's a minimal streaming chat loop:

# chat.py
import os
from dotenv import load_dotenv
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
 
def main():
    load_dotenv()
    llm = ChatOpenAI(model="gpt-4o-mini")
    
    print("Chat started. Type 'quit' or 'exit' to stop.\n")
    
    while True:
        user_input = input("You: ")
        
        if user_input.lower() in ["quit", "exit"]:
            print("Goodbye!")
            break
        
        print("Assistant: ", end="", flush=True)
        
        for chunk in llm.stream([HumanMessage(content=user_input)]):
            print(chunk.content, end="", flush=True)
        
        print("\n")
 
if __name__ == "__main__":
    main()

How this works:

1. while True:: Infinite loop for continuous conversation
2. input("You: "): Get user input from terminal
3. llm.stream([HumanMessage(...)]): Stream LLM response
4. Streaming output with special parameters:
   • end="": Don't add newline after each chunk (keeps output on same line)
   • flush=True: Force immediate output to terminal without buffering

Why [HumanMessage(content=user_input)]?

LangChain's chat models expect a list of messages, not a raw string. Each message has a role:

• HumanMessage: User input
• AIMessage: LLM response
• SystemMessage: Instructions for the LLM (covered in Chapter 4)

Even for a single user message, you pass a list: [HumanMessage(content="Hello")].

Key limitation - Single-turn conversations: This chat loop is intentionally stateless. Each request sends only the current message, not previous conversation history. This means:

• The LLM won't remember what you asked before
• Follow-up questions like "What about its population?" won't work after asking "What's the capital of France?"
• This is a fundamental LLM characteristic - they have no memory unless you explicitly provide context

Example of the limitation:

You: What's the capital of France?
Assistant: Paris.
You: What's its population?
Assistant: I don't have enough context. What city are you asking about?

The while True loop provides UX continuity (you can keep chatting), but each turn is independent. Coming in Chapter 8: We'll implement conversation memory by storing and re-sending message history with each request.

Running the Chat Loop

python chat.py

Example interaction:

Chat started. Type 'quit' or 'exit' to stop.
 
You: What is LangChain?
Assistant: LangChain is a framework for developing applications powered by language models. It provides tools for prompt management, chains, agents, and memory.
 
You: Give me a simple example
Assistant: Here's a basic example: ...
 
You: quit
Goodbye!

Understanding the Streaming API

What is a "chunk"?

Each chunk is an AIMessageChunk object with:

• content: The generated text tokens
• response_metadata: Model info, token counts, etc.

for chunk in llm.stream([HumanMessage(content="Hello")]):
    print(f"Chunk: {chunk}")
    print(f"Content: {chunk.content}")
    print(f"Type: {type(chunk)}")

Output:

Chunk: content='Hello' response_metadata={'model_provider': 'openai', ...}
Content: Hello
Type: <class 'langchain_core.messages.ai.AIMessageChunk'>
 
Chunk: content='!' response_metadata={...}
Content: !
Type: <class 'langchain_core.messages.ai.AIMessageChunk'>
 
Chunk: content=' How' response_metadata={...}
Content:  How
Type: <class 'langchain_core.messages.ai.AIMessageChunk'>

Accumulating the Full Response

Sometimes you need the complete response (for logging, testing, or further processing):

def chat_with_accumulation():
    load_dotenv()
    llm = ChatOpenAI(model="gpt-4o-mini")
    
    user_input = input("You: ")
    
    full_response = ""
    print("Assistant: ", end="", flush=True)
    
    for chunk in llm.stream([HumanMessage(content=user_input)]):
        print(chunk.content, end="", flush=True)
        full_response += chunk.content
    
    print("\n")
    
    # Now you have the complete response
    print(f"[DEBUG] Full response length: {len(full_response)} chars")
    return full_response

This pattern is common when you need to:

• Save the conversation to a database
• Parse the response for structured data
• Calculate token usage or costs

Your project structure after this section:

langchain-chat/
├── venv/
├── .env
├── .gitignore
├── requirements.txt
├── test_install.py
└── chat.py              # Streaming chat loop (new!)

Next: Section 3.4 shows how to handle different model types with smart parameter configuration.

3.4) Smart Config: Handling Parameters for Reasoning vs Chat Models

OpenAI offers two types of models with different capabilities and control mechanisms:

Chat models (gpt-4o, gpt-4o-mini):

• Fast and conversational
• Support temperature for controlling randomness and creativity
• Best for general tasks, creative writing, routine coding

Reasoning models (o1, o3, GPT-5):

• Slower but more logical and consistent
• Do NOT support temperature (use internal reasoning instead)
• Best for complex math, multi-step planning, formal analysis

The key difference: Chat models use probabilistic sampling (you control randomness), while reasoning models use deterministic internal logic (the model controls its own reasoning process).

Understanding Temperature (Chat Models Only)

What is temperature?

Temperature is a number between 0.0 and 2.0 that controls how creative the model's responses are. At low values (near 0), you get consistent, predictable answers. At high values (near 2.0), you get creative, varied responses. Think of it like a "creativity dial".

How it works: When generating each word, the model sees many possible next words with different probabilities. Temperature affects how the model chooses:

• Low temperature (0.0): Almost always picks the highest-probability word → consistent, focused responses
• High temperature (2.0): More likely to pick lower-probability words → diverse, creative responses

Important: Temperature only works with chat models (gpt-4o, gpt-4o-mini). It does not apply to reasoning models (GPT-5, o1, o3), which use internal logic instead of probabilistic sampling.

Temperature values guide:

• 0.0: Highly deterministic, focused, and consistent

  • Use for: factual Q&A, routine code generation, structured output
  • Same input → nearly identical output every time
  • Example: "What is 2+2?" → Always "4"

• 0.7–1.0: Standard sampling behavior (default is 1.0)

  • Use for: general conversation, explanations, balanced responses
  • Moderate variation in phrasing and examples
  • Example: "Explain photosynthesis" → Different wording each time, same core info

• 1.2–2.0: More creative and diverse, less predictable

  • Use for: creative writing, brainstorming, ideation
  • High variation in tone, structure, and wording
  • Example: "Write a poem about the moon" → Very different styles each time

Note: Values above 1.0 increase creativity but may reduce factual accuracy and coherence. Maximum value is 2.0.

Example: Temperature impact (chat models only)

# Temperature 0.0 - deterministic, same answer every time
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0.0)
response = llm.invoke([HumanMessage(content="What is 2+2?")])
print(response.content)  # Output: 4
 
# Temperature 1.0 - default behavior, slight variation possible
llm = ChatOpenAI(model="gpt-4o-mini", temperature=1.0)
response = llm.invoke([HumanMessage(content="What is 2+2?")])
print(response.content)  # Output: 4 (may include brief explanation)

For closed, factual questions, temperature has little effect on correctness.

For open-ended or creative tasks, temperature significantly influences diversity, tone, and style.

What If You Use Chat Model Parameters on Reasoning Models?

It depends on the model - some reject it, others silently ignore it:

# ❌ This will fail with o3 models
llm = ChatOpenAI(model="o3-mini", temperature=0.7)

Error:

BadRequestError: Temperature is not supported with this model

Different models, different policies:

• o1 / o3 models: Explicitly reject unsupported parameters. If temperature is included, the API returns a 400 BadRequest error immediately.
• GPT-5 models: More permissive - the parameter is accepted but silently ignored. Your request succeeds, but temperature has no effect.

Why this matters: Always check which model you're using and configure parameters accordingly. Using the wrong parameters can either cause errors or silently fail, wasting debugging time.

How to Control Reasoning Model Behavior

Now you know chat models use temperature and reasoning models don't. So how do you control reasoning models?

Reasoning models are tuned through prompt design, not parameters:

• Reasoning models do not expose temperature or similar controls
• Instead, you guide behavior by how you write the prompt:
  • Explicit instructions: "Think step-by-step", "Show your work"
  • Constraints as rules: "You must not assume...", "Always verify..."
  • Structured requirements: "Output in JSON format", "Include reasoning before answer"
  • Decision logic: "If condition A, then do X, otherwise do Y"

Example: Chat parameters vs Reasoning prompts

# ❌ Chat approach - won't work with reasoning models
llm = ChatOpenAI(model="o3-mini", temperature=0.5)
# Error: BadRequestError: Temperature is not supported
 
# ✅ Reasoning approach - guide through prompt structure
prompt = """
Solve this problem step-by-step:
1. State what you know
2. Show your calculations
3. Verify your answer
 
Problem: If x + 5 = 12, what is x?
"""
llm = ChatOpenAI(model="o3-mini")
response = llm.invoke([HumanMessage(content=prompt)])
print(response.content)

Output:

1. What I know: x + 5 = 12
2. Calculations: x = 12 - 5 = 7
3. Verification: 7 + 5 = 12 ✓
 
Answer: x = 7

Key insight: Chat models are controlled by parameters, reasoning models are controlled by prompts.

Model Selection Decision Table

Now that you understand how to control both types of models, here's when to use each:

Cost and Latency Trade-offs

Understanding the practical trade-offs helps you choose the right model for your use case:

Notes:

• Speed reflects typical response latency (varies by prompt length and complexity)
• Cost is a relative comparison - check current pricing on OpenAI's website
• Reasoning models trade speed and cost for consistency and correctness
• Chat models prioritize responsiveness and efficiency

When to use reasoning models (GPT-5, o1, o3):

• Multi-step math and STEM(Science, Technology, Engineering, Mathematics) problems requiring correct intermediate steps
• Complex logical analysis with dependencies and constraints
• Code debugging with multiple interacting causes
• Planning tasks with many rules, edge cases, or trade-offs
• Agent workflows requiring consistency and long-horizon thinking

When to use chat models (gpt-4o, gpt-4o-mini):

• General conversation and interactive chat
• Simple Q&A with limited reasoning depth
• Content generation (blogs, summaries, creative writing)
• Routine code generation and boilerplate tasks
• Applications where speed and cost matter more than deep reasoning

Next: Section 3.5 shows debugging techniques to inspect what's actually sent to the LLM.

3.5) Debugging: Inspecting Responses and Token Usage

When your LLM behaves unexpectedly, you need to see exactly what was sent and received. This section shows how to inspect LLM calls and debug issues.

Why Debugging Matters

Common debugging scenarios:

• "Why did the LLM give this answer?" → Check the exact prompt
• "How much did this request cost?" → Check token usage
• "Why is this so slow?" → Measure latency
• "Is my message formatting correct?" → Inspect the message structure

The challenge: When you call llm.invoke(), you get a response object. But what's actually in it? What information is available for debugging?

Understanding the Response Object

Before debugging, you need to understand what llm.invoke() returns.

Basic structure:

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
 
llm = ChatOpenAI(model="gpt-4o-mini")
response = llm.invoke([HumanMessage(content="Hello")])
 
# What's in the response?
print(type(response))  # AIMessage
print(response.content)  # The actual text
print(response.response_metadata)  # Token usage, model info, etc.

Output:

<class 'langchain_core.messages.ai.AIMessage'>
Hello! How can I assist you today?
{
  'token_usage': {
    'completion_tokens': 9,
    'prompt_tokens': 8,
    'total_tokens': 17
  },
  'model_name': 'gpt-4o-mini-2024-07-18',
  'finish_reason': 'stop',
  ...
}

Key parts of the response:

• response.content: The text the LLM generated
• response.response_metadata: Dictionary with:
  • token_usage: How many tokens were used (for cost calculation)
  • model_name: Exact model version that responded
  • finish_reason: Why generation stopped (see Debug Mode section for details)

Accessing token usage:

token_usage = response.response_metadata['token_usage']
print(f"Prompt tokens: {token_usage['prompt_tokens']}")
print(f"Response tokens: {token_usage['completion_tokens']}")
print(f"Total: {token_usage['total_tokens']}")

Output:

Prompt tokens: 8
Response tokens: 9
Total: 17

Why this matters: You need these values for debugging, cost tracking, and optimizing your prompts.

Calculating Costs from Token Usage

Token usage determines cost. Each model has different pricing:

GPT-4o-mini (as of January 2026):

• Input: $0.15 per 1M tokens
• Output: $0.60 per 1M tokens

GPT-4o:

• Input: $2.50 per 1M tokens
• Output: $10.00 per 1M tokens

Cost calculation function:

def calculate_cost(token_usage, model_name):
    """Calculate cost based on token usage."""
    prompt_tokens = token_usage.get('prompt_tokens', 0)
    completion_tokens = token_usage.get('completion_tokens', 0)
    
    # Pricing per 1M tokens (as of January 2026)
    pricing = {
        'gpt-4o-mini': {'input': 0.15, 'output': 0.60},
        'gpt-4o': {'input': 2.50, 'output': 10.00},
        'gpt-5': {'input': 1.25, 'output': 10.00},
    }
    
    if model_name not in pricing:
        return None
    
    input_cost = (prompt_tokens / 1_000_000) * pricing[model_name]['input']
    output_cost = (completion_tokens / 1_000_000) * pricing[model_name]['output']
    
    return input_cost + output_cost
 
# Example
response = llm.invoke([HumanMessage(content="Explain quantum computing")])
token_usage = response.response_metadata['token_usage']
cost = calculate_cost(token_usage, "gpt-4o-mini")
print(f"Cost: ${cost:.6f}")

Output:

Cost: $0.000123

Why this matters: Production apps can handle 50,000+ requests/day. At $0.002 per request, that's $3,000/month. Use the wrong model or bloated prompts, and costs jump to $30,000/month. A retry loop bug can burn thousands overnight. Track token usage from day one.

Enabling Debug Mode (When You Need Raw API Details)

The response object and custom wrapper handle most debugging needs. But sometimes you need to see exactly what LangChain sends to OpenAI - the raw JSON request and response.

When you might need this:

• Debugging LangChain's message formatting
• Verifying API parameters are set correctly
• Investigating unexpected API errors
• Understanding the exact API payload

LangChain has built-in debug logging via langchain_core.globals:

from langchain_core.globals import set_debug
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
 
set_debug(True)
 
# Now all LLM calls will print debug info
llm = ChatOpenAI(model="gpt-4o-mini")
response = llm.invoke([HumanMessage(content="Hello")])

Output:

[llm/start] [llm:ChatOpenAI] Entering LLM run with input:
{
  "prompts": [
    "Human: Hello"
  ]
}
[llm/end] [llm:ChatOpenAI] [1.45s] Exiting LLM run with output:
{
  "generations": [
    [
      {
        "text": "Hello! How can I assist you today?",
        "generation_info": {
          "finish_reason": "stop",
          "logprobs": null
        },
        "type": "ChatGeneration",
        ...
      }
    ]
  ],
  "llm_output": {
    "token_usage": {
      "completion_tokens": 9,
      "prompt_tokens": 8,
      "total_tokens": 17,
      ...
    },
    "model_provider": "openai",
    "model_name": "gpt-4o-mini-2024-07-18",
    ...
  },
}

Note: Output format varies by LLM provider. This example shows OpenAI's structure.

What the debug output reveals:

Debug mode shows the complete LangChain → OpenAI communication flow:

1. Message format transformation:

# Your code
[HumanMessage(content="Hello")]
 
# What you see in debug output
{
  "prompts": ["Human: Hello"]
}

Debug mode shows how LangChain represents your message internally before sending to the LLM.

2. Generation completion status:

"finish_reason": "stop"

Why the generation ended:

• "stop": The model completed the response naturally
• "length": The response was cut off because it reached the max_tokens limit
• "tool_calls": The model ended the generation by producing tool call instructions instead of a final text response (Chapter 12)
• "content_filter": The response was blocked or suppressed due to safety or content moderation rules

If you see "length", increase max_tokens to get the full response.

3. Token usage breakdown:

"token_usage": {
  "completion_tokens": 9,
  "prompt_tokens": 8,
  "total_tokens": 17,
  "completion_tokens_details": {
    "reasoning_tokens": 0  # For reasoning models (o1/o3, etc.)
  },
  "prompt_tokens_details": {
    "cached_tokens": 0  # Prompt caching (saves costs)
  }
}

Beyond basic counts, you can see:

• reasoning_tokens: Internal reasoning steps (only for reasoning models)
• cached_tokens: How many prompt tokens were served from cache (reduces cost)

4. Model version and fingerprint:

"model_name": "gpt-4o-mini-2024-07-18",
"system_fingerprint": "fp_8bbc38b4db"

• model_name: Exact snapshot version (explains why responses change over time)
• system_fingerprint: OpenAI's backend configuration ID (changes when they update systems)

5. Request timing:

[llm/end] [llm:ChatOpenAI] [1.56s]

The [1.45s] shows total request duration—useful for identifying slow queries.

Next: Section 3.6 shows how to handle common errors gracefully.

3.6) Handling Failures (Simulate and fix common errors)

Production LLM applications face predictable failure modes: missing credentials, network timeouts, rate limits, and invalid inputs. This section shows you how to handle these errors gracefully and build robust applications from day one.

The Six Common Errors

1. Missing API Key

When this happens: You try to create a ChatOpenAI instance, but OPENAI_API_KEY is not set in your environment.

Example:

# .env file doesn't exist, or OPENAI_API_KEY is not defined
llm = ChatOpenAI(model="gpt-4o-mini")
response = llm.invoke([HumanMessage(content="Hello")])

Error you'll see:

OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable

How to fix:

1. Check your .env file exists in the project root
2. Verify the key name is exactly OPENAI_API_KEY (common typo: OPENAPI_KEY)
3. Ensure load_dotenv() is called before creating the LLM

2. Wrong API Key

When this happens: Your .env file contains an invalid, expired, or incorrectly copied API key.

Example:

# .env has: OPENAI_API_KEY=sk-invalid-key-12345
llm = ChatOpenAI(model="gpt-4o-mini")
response = llm.invoke([HumanMessage(content="Hello")])

Error you'll see:

AuthenticationError: Incorrect API key provided

How to fix:

1. Go to https://platform.openai.com/api-keys
2. Verify your key is still active (not revoked or expired)
3. Generate a new key if needed
4. Copy the entire key carefully (common mistake: missing first/last characters)
5. Paste into .env with no extra spaces:

OPENAI_API_KEY=sk-proj-exactkeyhere

3. Network Failures

When this happens: Your internet connection drops, or OpenAI's servers are temporarily unreachable during a request.

Example:

# WiFi disconnects mid-request, or OpenAI API is down
response = llm.invoke([HumanMessage(content="Hello")])

Error you'll see:

APIConnectionError: Connection error

How to fix:

1. Check your internet connection
2. Verify OpenAI status at https://status.openai.com

4. Rate Limits

When this happens: You send too many requests in a short time and exceed your API quota.

Example:

# Sending 1000 requests instantly
for i in range(1000):
    llm.invoke([HumanMessage(content=f"Request {i}")])

Error you'll see:

RateLimitError: Rate limit reached for requests

How to fix:

1. Check your rate limits at https://platform.openai.com/account/limits
2. Upgrade your plan if you need higher limits
3. Use batch processing for large workloads (covered in Chapter 6)

5. Invalid Model Name

When this happens: You specify a model name that doesn't exist or isn't available in your plan.

Example:

llm = ChatOpenAI(model="gpt-99-ultra")  # Doesn't exist
response = llm.invoke([HumanMessage(content="Hello")])

Error you'll see:

NotFoundError: The model `gpt-99-ultra` does not exist or you do not have access to it

How to fix:

1. Check available models in your plan at https://platform.openai.com/docs/models

6. Token Limit Exceeded

When this happens: Your prompt is too long and exceeds the model's maximum context window.

Example:

# Creating a 1 million character prompt
huge_prompt = "x" * 1_000_000
response = llm.invoke([HumanMessage(content=huge_prompt)])

Error you'll see:

BadRequestError: This model's maximum context length is 128000 tokens. However, your messages resulted in 250000 tokens.

How to fix:

1. Check input length before sending
2. Know your model's limits:
   • gpt-4o-mini: 128K tokens
   • gpt-4o: 128K tokens
   • gpt-5: 400K tokens
3. For long documents, use chunking or summarization (covered in Chapter 9)

Next steps: Chapter 4 shows how to design reusable prompt templates that separate prompt engineering from application code.