Iterators and the Iterator Protocol: How __iter__ and __next__ Really Work

Iterables vs. iterators

These two words look similar and get used interchangeably in casual conversation, but they mean different things in Python and the distinction explains a lot of otherwise confusing behaviour. An iterable is any object that knows how to produce an iterator — lists, tuples, strings, dicts, sets, and files all qualify. An iterator is the object that actually keeps track of where you are in the sequence and produces the next value on demand.

A list is iterable, but a list is not itself an iterator:

numbers = [10, 20, 30]
it = iter(numbers)
print(type(numbers))   # <class 'list'>
print(type(it))        # <class 'list_iterator'>

Calling iter() on the list produces a separate list_iterator object. This is why you can loop over the same list twice with two independent for loops without one interfering with the other — each loop asks for its own fresh iterator.

__iter__ and __next__

The protocol has exactly two parts. An iterable defines __iter__, which returns an iterator. An iterator defines both __iter__ (which conventionally returns itself) and __next__, which returns the next value or raises StopIteration when there is nothing left:

numbers = [10, 20, 30]
it = iter(numbers)          # calls numbers.__iter__()
print(next(it))             # calls it.__next__() -> 10
print(next(it))             # 20
print(next(it))             # 30
print(next(it))             # raises StopIteration

Because an iterator's __iter__ returns itself, an iterator is also iterable — you can pass an iterator directly to a for loop and it will work exactly as you would expect, just without producing a second, independent copy of the sequence.

Driving an iterator by hand

You rarely need to call next() directly in application code, but doing it once makes the whole protocol click. The built-in next() function accepts an optional default, which is useful for treating exhaustion as a normal case rather than an error:

it = iter([1, 2])
print(next(it, "done"))   # 1
print(next(it, "done"))   # 2
print(next(it, "done"))   # done -- no StopIteration raised

Without the default, that third call would raise StopIteration, which is exactly the signal a for loop relies on internally.

Building a custom iterator class

Implementing the protocol yourself is a useful exercise for understanding lazy evaluation, and occasionally the right tool when you need an iterator that also exposes extra methods or state a generator cannot conveniently offer. Here is a small iterator that counts down from a starting number:

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

for n in Countdown(3):
    print(n)   # 3, 2, 1

Every piece of state the object needs — here just self.current — lives on the instance, so the object remembers exactly where it left off between calls to __next__. This is the same underlying idea a generator handles automatically for you, just written out explicitly.

StopIteration and the for loop

It is worth spelling out what a for loop is doing, because the mechanism is completely ordinary Python that you could write yourself with a while loop:

iterator = iter(some_iterable)
while True:
    try:
        item = next(iterator)
    except StopIteration:
        break
    # loop body
    print(item)

A for loop is exactly this pattern, written more concisely. StopIteration is not a failure condition to be alarmed about — it is the normal, expected signal that an iterator is exhausted, and Python's loop machinery catches it silently every single time a loop finishes.

Iterators vs. generators

A generator function — one containing yield — is simply a convenient way to get an object that satisfies the iterator protocol without writing a class. When you call a generator function, Python automatically builds an object with working __iter__ and __next__ methods, and each yield becomes a return value from __next__:

def countdown(start):
    current = start
    while current > 0:
        yield current
        current -= 1

for n in countdown(3):
    print(n)   # 3, 2, 1

This produces the same sequence as the Countdown class above with far less code, because the generator machinery keeps track of the paused execution state for you instead of requiring you to manage an instance attribute by hand. Reach for a generator by default; write a full iterator class only when you need extra methods, properties, or behaviour that a plain generator function cannot express.