35. How Iteration Works: Iterables and Iterators

Throughout this book, you've been using for loops to iterate over lists, strings, dictionaries, and other collections. You've written code like for item in my_list: countless times. But what actually happens behind the scenes when Python executes a for loop? How does Python know how to step through different types of collections?

In this chapter, we'll explore Python's iteration protocol—the mechanism that makes for loops work. You'll learn about iterables (objects you can loop over) and iterators (objects that do the actual stepping through values). Understanding this distinction will deepen your knowledge of how Python works and prepare you for working with generators in Chapter 36.

35.1) What It Means for an Object to Be Iterable

35.1.1) The Concept of Iterability

An iterable is any Python object that can be looped over with a for loop. When we say "looped over," we mean Python can retrieve items from the object one at a time, in sequence.

You've already worked with many iterables:

python

# Lists are iterable
numbers = [1, 2, 3, 4, 5]
for num in numbers:
    print(num)  # Output: 1, 2, 3, 4, 5 (on separate lines)
 
# Strings are iterable
text = "Python"
for char in text:
    print(char)  # Output: P, y, t, h, o, n (on separate lines)
 
# Dictionaries are iterable (over keys by default)
student = {"name": "Alice", "age": 20, "grade": "A"}
for key in student:
    print(key)  # Output: name, age, grade (on separate lines)

All these objects—lists, strings, dictionaries, tuples, sets, ranges, and files—are iterables because they support Python's iteration protocol (a set of rules that allows Python to loop over them).

35.1.2) What Makes an Object Iterable

For an object to be iterable, it must implement a special method called __iter__(). This method returns an iterator object. Don't worry about the details yet—we'll explore iterators in the next section.

You can check if an object is iterable by trying to get an iterator from it using the built-in iter() function:

python

# Testing if objects are iterable
numbers = [1, 2, 3]
iterator = iter(numbers)  # Works - lists are iterable
print(type(iterator))  # Output: <class 'list_iterator'>
 
text = "Hello"
iterator = iter(text)  # Works - strings are iterable
print(type(iterator))  # Output: <class 'str_iterator'>
 
# Trying with a non-iterable object
value = 42
try:
    iterator = iter(value)  # Fails - integers are not iterable
except TypeError as e:
    print(f"Error: {e}")  # Output: Error: 'int' object is not iterable

When you call iter() on an iterable object, Python calls the object's __iter__() method and returns an iterator. If the object doesn't have this method, you get a TypeError.

35.1.3) Iterables vs Sequences

It's important to understand that not all iterables are sequences. A sequence is a specific type of iterable that supports indexing and has a defined order.

python

# Sequences support indexing
my_list = [10, 20, 30]
print(my_list[0])  # Output: 10
 
my_string = "Python"
print(my_string[2])  # Output: t
 
# Sets are iterable but NOT sequences (no indexing, no guaranteed order)
my_set = {1, 2, 3}
for item in my_set:
    print(item)  # Works - sets are iterable
 
# But indexing doesn't work
try:
    print(my_set[0])  # Fails - sets don't support indexing
except TypeError as e:
    print(f"Error: {e}")  # Output: Error: 'set' object is not subscriptable

Key distinction: All sequences (lists, tuples, strings, ranges) are iterables, but not all iterables are sequences. Sets and dictionaries are iterables but not sequences because they don't support indexing.

35.1.4) Why Iterability Matters

Understanding iterability helps you:

Know what you can loop over: Any iterable works with for loops
Understand error messages: "object is not iterable" means you can't use it in a for loop
Use comprehensions: List, set, and dictionary comprehensions work with any iterable
Work with built-in functions: Many built-ins like sum(), max(), min(), and sorted() accept any iterable

python

# All these work because they accept iterables
numbers = [1, 2, 3, 4, 5]
print(sum(numbers))  # Output: 15
 
text = "Python"
print(max(text))  # Output: y (highest alphabetically)
 
# Even works with sets
unique_values = {10, 5, 20, 15}
print(sorted(unique_values))  # Output: [5, 10, 15, 20]

35.2) Everyday Iterators in Python (Files, Ranges, Dictionaries, and More)

35.2.1) What Is an Iterator

An iterator is an object that represents a stream of data. It returns one value at a time when you ask for the next item. Once an iterator has returned all its values, it's exhausted and can't be reused.

Think of an iterator like a bookmark in a book:

It remembers where you are in the sequence
You can ask for the next item
Once you reach the end, you can't go back without creating a new iterator

The key difference between an iterable and an iterator:

An iterable is something you can iterate over (like a list)
An iterator is the object that does the iterating (the mechanism that steps through the list)

python

# A list is an iterable
numbers = [1, 2, 3]
 
# Getting an iterator from the iterable
iterator = iter(numbers)
 
# The iterator is a separate object
print(type(numbers))    # Output: <class 'list'>
print(type(iterator))   # Output: <class 'list_iterator'>

35.2.2) Iterators in for Loops

When you write a for loop, Python automatically creates an iterator behind the scenes:

python

numbers = [10, 20, 30]
 
# What you write:
for num in numbers:
    print(num)
 
# What Python does internally (conceptually):
# 1. Call iter(numbers) to get an iterator
# 2. Repeatedly call next() on the iterator
# 3. Stop when the iterator raises StopIteration

Here's what that looks like explicitly:

python

numbers = [10, 20, 30]
 
# Manual iteration (what for does automatically)
iterator = iter(numbers)
try:
    print(next(iterator))  # Output: 10
    print(next(iterator))  # Output: 20
    print(next(iterator))  # Output: 30
    print(next(iterator))  # Would raise StopIteration
except StopIteration:
    print("No more items")  # Output: No more items

The for loop handles the StopIteration exception automatically, which is why you never see it in normal code.

35.2.3) File Objects as Iterators

File objects are excellent examples of iterators. When you iterate over a file, it reads one line at a time:

python

# Create a sample file
with open("students.txt", "w") as file:
    file.write("Alice\n")
    file.write("Bob\n")
    file.write("Charlie\n")
 
# Reading the file line by line
with open("students.txt", "r") as file:
    for line in file:
        print(line.strip())  # Output: Alice, Bob, Charlie (on separate lines)

File objects are both iterable and iterators. They return themselves when you call iter() on them:

python

with open("students.txt", "r") as file:
    iterator = iter(file)
    print(file is iterator)  # Output: True (same object)
    
    # Reading lines manually
    print(next(iterator))  # Output: Alice
    print(next(iterator))  # Output: Bob
    print(next(iterator))  # Output: Charlie

This is memory-efficient because Python doesn't load the entire file into memory—it reads one line at a time as you request it.

35.2.4) Range Objects as Iterators

Range objects are iterables that generate numbers on demand:

python

# A range is an iterable
numbers = range(1, 4)
print(type(numbers))  # Output: <class 'range'>
 
# Getting an iterator from the range
iterator = iter(numbers)
print(type(iterator))  # Output: <class 'range_iterator'>
 
# Using the iterator
print(next(iterator))  # Output: 1
print(next(iterator))  # Output: 2
print(next(iterator))  # Output: 3

Ranges are memory-efficient because they don't store all numbers in memory—they calculate each number when requested:

python

# This range represents 1 million numbers but uses minimal memory
large_range = range(1000000)
print(type(large_range))  # Output: <class 'range'>
 
# Getting an iterator
iterator = iter(large_range)
print(next(iterator))  # Output: 0
print(next(iterator))  # Output: 1
# ... can continue for 1 million values

35.2.5) Dictionary Iterators

Dictionaries provide different iterators for keys, values, and items:

python

student = {"name": "Alice", "age": 20, "grade": "A"}
 
# Iterating over keys (default)
for key in student:
    print(key)  # Output: name, age, grade (on separate lines)
 
# Explicitly getting a keys iterator
keys_iterator = iter(student.keys())
print(next(keys_iterator))  # Output: name
print(next(keys_iterator))  # Output: age
 
# Iterating over values
values_iterator = iter(student.values())
print(next(values_iterator))  # Output: Alice
print(next(values_iterator))  # Output: 20
 
# Iterating over items (key-value pairs)
items_iterator = iter(student.items())
print(next(items_iterator))  # Output: ('name', 'Alice')
print(next(items_iterator))  # Output: ('age', 20)

35.2.6) Iterators Are Exhaustible

A crucial property of iterators is that they can only be used once. Once exhausted, they don't reset:

python

numbers = [1, 2, 3]
iterator = iter(numbers)
 
# First pass through the iterator
print(next(iterator))  # Output: 1
print(next(iterator))  # Output: 2
print(next(iterator))  # Output: 3
 
# Iterator is now exhausted
try:
    print(next(iterator))  # Raises StopIteration
except StopIteration:
    print("Iterator exhausted")  # Output: Iterator exhausted
 
# To iterate again, create a new iterator
iterator = iter(numbers)
print(next(iterator))  # Output: 1 (fresh start)

This is different from the iterable itself, which can be iterated over multiple times:

python

numbers = [1, 2, 3]
 
# First iteration
for num in numbers:
    print(num)  # Output: 1, 2, 3
 
# Second iteration (works fine - creates a new iterator)
for num in numbers:
    print(num)  # Output: 1, 2, 3

35.3) Using iter() and next() to Step Through Iterables

35.3.1) The iter() Function

The iter() function takes an iterable and returns an iterator. This is the first step in the iteration protocol:

python

# Creating iterators from different iterables
numbers = [10, 20, 30]
iterator = iter(numbers)
print(type(iterator))  # Output: <class 'list_iterator'>
 
text = "Hi"
text_iterator = iter(text)
print(type(text_iterator))  # Output: <class 'str_iterator'>
 
my_set = {1, 2, 3}
set_iterator = iter(my_set)
print(type(set_iterator))  # Output: <class 'set_iterator'>

Each type of iterable returns its own specialized iterator type, but they all work the same way—you call next() to get the next value.

35.3.2) The next() Function

The next() function retrieves the next item from an iterator. When there are no more items, it raises StopIteration:

python

colors = ["red", "green", "blue"]
iterator = iter(colors)
 
# Getting items one at a time
print(next(iterator))  # Output: red
print(next(iterator))  # Output: green
print(next(iterator))  # Output: blue
 
# No more items
try:
    print(next(iterator))  # Raises StopIteration
except StopIteration:
    print("No more colors")  # Output: No more colors

35.3.3) Providing a Default Value to next()

You can provide a default value to next() as a second argument. When the iterator is exhausted, instead of raising a StopIteration exception, next() will return the default value you specified:

python

numbers = [1, 2, 3]
iterator = iter(numbers)
 
print(next(iterator))           # Output: 1
print(next(iterator))           # Output: 2
print(next(iterator))           # Output: 3
print(next(iterator, "Done"))   # Output: Done (default value, no exception)
print(next(iterator, "Done"))   # Output: Done (still exhausted)

This is useful when you want to handle the end of iteration gracefully without exception handling:

35.4) Creating Custom Iterators with iter and next

35.4.1) Why Create Custom Iterators

Python's built-in iterables (lists, strings, files) cover most common cases. However, sometimes you need to create your own iterable objects for specialized behavior:

Generating sequences with custom logic
Iterating over data structures you design
Creating memory-efficient iteration over large datasets
Implementing lazy evaluation (computing values only when needed)

Creating a custom iterator requires implementing two special methods: __iter__() and __next__().

35.4.2) The Iterator Protocol

To make an object an iterator, it must implement:

__iter__(): Returns the iterator object itself (usually self)
__next__(): Returns the next value in the sequence, or raises StopIteration when done

python

class SimpleCounter:
    """An iterator that counts from start to end."""
    
    def __init__(self, start, end):
        self.current = start
        self.end = end
    
    def __iter__(self):
        """Return the iterator object (self)."""
        return self
    
    def __next__(self):
        """Return the next value or raise StopIteration."""
        if self.current > self.end:
            raise StopIteration
        
        value = self.current
        self.current += 1
        return value
 
# Using the custom iterator
counter = SimpleCounter(1, 5)
 
for num in counter:
    print(num)
# Output: 1
# Output: 2
# Output: 3
# Output: 4
# Output: 5

Let's break down what happens:

The for loop calls iter(counter), which calls counter.__iter__() and gets back counter itself
The loop repeatedly calls next(counter), which calls counter.__next__()
Each call to __next__() returns the next number and increments current
When current > end, __next__() raises StopIteration, and the loop stops

35.4.3) Manual Use of Custom Iterators

You can also use custom iterators manually with iter() and next():

python

counter = SimpleCounter(10, 13)
 
# Get the iterator (returns itself)
iterator = iter(counter)
print(iterator is counter)  # Output: True
 
# Get values manually
print(next(iterator))  # Output: 10
print(next(iterator))  # Output: 11
print(next(iterator))  # Output: 12
print(next(iterator))  # Output: 13
 
# Now exhausted
try:
    print(next(iterator))
except StopIteration:
    print("Counter exhausted")  # Output: Counter exhausted

35.4.4) Iterators Are Exhaustible (Revisited)

Remember that iterators can only be used once:

python

counter = SimpleCounter(1, 3)
 
# First iteration
for num in counter:
    print(num)  # Output: 1, 2, 3
 
# Second iteration (doesn't work - iterator is exhausted)
for num in counter:
    print(num)  # Nothing printed - iterator is already exhausted

To iterate again, you need to create a new instance:

python

# Create a new counter for each iteration
for num in SimpleCounter(1, 3):
    print(num)  # Output: 1, 2, 3
 
for num in SimpleCounter(1, 3):
    print(num)  # Output: 1, 2, 3 (new iterator)

35.4.5) Creating an Iterable Class (Not Just an Iterator)

Often, you want a class that's iterable but creates a fresh iterator each time. To do this, separate the iterable from the iterator:

python

class CounterIterable:
    """An iterable that creates fresh counter iterators."""
    
    def __init__(self, start, end):
        self.start = start
        self.end = end
    
    def __iter__(self):
        """Return a new iterator each time."""
        return CounterIterator(self.start, self.end)
 
class CounterIterator:
    """The actual iterator that does the counting."""
    
    def __init__(self, start, end):
        self.current = start
        self.end = end
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.current > self.end:
            raise StopIteration
        value = self.current
        self.current += 1
        return value
 
# Now we can iterate multiple times
counter = CounterIterable(1, 3)
 
# First iteration
for num in counter:
    print(num)  # Output: 1, 2, 3
 
# Second iteration (works because __iter__ creates a new iterator)
for num in counter:
    print(num)  # Output: 1, 2, 3

This pattern separates concerns:

CounterIterable is the iterable—it knows how to create iterators
CounterIterator is the iterator—it knows how to step through values

35.4.6) Practical Example: Iterating Over a Custom Data Structure

Let's create an iterator for a custom data structure—a simple playlist:

python

class Playlist:
    """A music playlist that can be iterated over."""
    
    def __init__(self):
        self.songs = []
    
    def add_song(self, title, artist):
        """Add a song to the playlist."""
        self.songs.append({"title": title, "artist": artist})
    
    def __iter__(self):
        """Return an iterator for the playlist."""
        return PlaylistIterator(self.songs)
 
class PlaylistIterator:
    """Iterator for stepping through songs in a playlist."""
    
    def __init__(self, songs):
        self.songs = songs
        self.index = 0
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.index >= len(self.songs):
            raise StopIteration
        
        song = self.songs[self.index]
        self.index += 1
        return song
 
# Using the playlist
playlist = Playlist()
playlist.add_song("Imagine", "John Lennon")
playlist.add_song("Bohemian Rhapsody", "Queen")
playlist.add_song("Hotel California", "Eagles")
 
# Iterate over songs
print("Now playing:")
for song in playlist:
    print(f"  {song['title']} by {song['artist']}")
# Output: Now playing:
# Output:   Imagine by John Lennon
# Output:   Bohemian Rhapsody by Queen
# Output:   Hotel California by Eagles
 
# Can iterate again (creates a new iterator)
print("\nReplay:")
for song in playlist:
    print(f"  {song['title']}")
# Output: Replay:
# Output:   Imagine
# Output:   Bohemian Rhapsody
# Output:   Hotel California

35.4.7) When to Use Custom Iterators

Create custom iterators when:

You need lazy evaluation: Generate values on demand rather than storing them all
You have a custom data structure: Make it iterable so it works with for loops
You need special iteration logic: Skip items, transform values, or implement complex stepping
Memory efficiency matters: Generate large sequences without storing them

However, in Chapter 36, you'll learn about generators, which provide a much simpler way to create iterators using the yield keyword. Generators are usually preferred over manually implementing __iter__() and __next__() because they're more concise and easier to understand.

Understanding how to create custom iterators gives you insight into how Python's iteration protocol works, even if you'll often use generators instead. The concepts you've learned here—__iter__(), __next__(), and StopIteration—are fundamental to understanding generators and other advanced iteration techniques in the next chapter.