Iterators & Generators
Find out what for really does, make your own objects loopable, and use yield and generator expressions to process data lazily, one item at a time.
You will learn
- The iterator protocol:
iter(),next()andStopIteration. - The difference between an iterable and an iterator, and why iterators get "used up".
- How to write an iterator class with
__iter__and__next__. - Generator functions with
yieldandyield from. - Generator expressions, lazy pipelines, and a few
itertoolshelpers.
How the for loop really works
Behind the scenes, for x in thing: does three things:
- It calls
iter(thing)to get an iterator: an object that remembers where it is. - It calls
next(iterator)repeatedly, and each call returns the next item. - When there are no more items,
next()raisesStopIteration, and the loop ends quietly.
Analogy: an iterable is a book, and an iterator is a bookmark in it. You can have many bookmarks in one book. A bookmark only moves forward, and once it reaches the end, it stays there.
| Iterable | Iterator | |
|---|---|---|
| Has | __iter__() that returns a new iterator | __next__(), and __iter__() that returns itself |
| Examples | list, str, dict, set, range, file | what iter(list) returns; generators; zip, map, enumerate objects; files |
| Loop over it twice? | ✅ yes (each loop gets a fresh iterator) | ❌ no (the second loop sees nothing) |
Iterators get used up
❌ The second loop is empty
✅ Store a list if you need it twice
☕ Java: __iter__ is Iterable.iterator(), and __next__ combines hasNext() and next(): instead of asking first, you call it, and it raises StopIteration when it's done. 🟨 JS: this is the same idea as [Symbol.iterator]() and {done, value}.
Writing an iterator class
To make your own class loopable, give it __iter__. The simplest approach is an object that is its own iterator, with __iter__ returning self and __next__ producing the items:
This works, but it is a lot of ceremony, and a Reverse object can only be looped over once. Generators make the same thing far shorter.
Generators
A generator function looks like a normal function, but it uses yield instead of return to hand back values. Calling it doesn't run the body. Instead it returns a generator object, which is an iterator. Each next() runs the body until the next yield, then pauses there and keeps every local variable as it was.
Python creates __iter__ and __next__ for you, saves the state between calls, and raises StopIteration when the function ends. Here is the pausing made visible:
The third call resumed after the last yield, finished the loop, printed done, and reached the end of the function. That means "no more items", so next() returned the default. Without a default it would raise StopIteration.
Why generators matter: laziness
A generator produces items on demand. It never builds the whole sequence in memory, so it can handle huge files, or even infinite sequences.
yield from
yield from iterable yields every item of another iterable. It is handy for delegating to sub-generators, or for walking recursive structures.
Generator expressions
A generator expression is a comprehension written with parentheses instead of brackets. It produces items lazily instead of building a list. When it is the only argument of a function, you can drop the extra parentheses.
List comprehension […] | Generator expression (…) | |
|---|---|---|
| Builds | a full list in memory, immediately | nothing up front; one item at a time |
| Reusable? | yes | no (it is used up after one pass) |
len() and indexing? | yes | no |
| Best for | results you'll keep or reuse | feeding sum, max, any, all, "".join, or a for loop once |
📝 New in 3.15: generator expressions accept * unpacking too (PEP 798), so sum(*row for row in matrix) sums every number in a list of lists.
⚠️ Pitfall: any() and all() short-circuit. With a generator, they stop consuming items as soon as the answer is known, and the items they consumed are gone. That is efficient, but surprising if you reuse the generator afterwards.
Lazy pipelines
Generators chain together like a factory line. Each stage pulls one item at a time from the stage before, so even a multi-gigabyte log file uses almost no memory.
A few itertools helpers
The itertools module is a toolbox of fast iterator building blocks.
Bonus: context managers from generators
Chapter 10 promised this one. contextlib.contextmanager turns a generator with exactly one yield into something you can use with with. The code before the yield is the set-up, and the code after it is the clean-up.
More examples
1. Reading a big file in fixed-size records
2. Fibonacci forever, taken lazily
3. A reusable iterable class (each loop gets a fresh generator)
4. Does any expense exceed the limit? (short-circuit)
5. Running totals with a generator
Recap
forcallsiter(), thennext()untilStopIteration.- An iterable can be looped over many times. An iterator is single-use, so convert it with
list()if you need it twice. - An iterator class needs
__iter__(returningself) and__next__. A generator function withyielddoes all of that for you. (expr for x in xs)is lazy, which is perfect forsum,any,all,maxandjoin. 3.15 adds*unpacking.- Chain generators into memory-light pipelines. Use
itertoolsfor counting, slicing, chaining, grouping and batching.
Exercises
Exercise 1 🟢 easy predict the output
What does this print?
Show solution
1 2, then [3], then []. The iterator remembers its position, and once it is exhausted it stays empty.
Exercise 2 🟢 easy build it
Write a generator evens(limit) that yields 0, 2, 4, … up to but not including limit.
Show solution
Each yield pauses the function, and the next next() continues from that point.
Exercise 3 🟢 easy refactor it
Make this use a generator expression, so that no temporary list is built: total = sum([len(w) for w in words])
Show solution
Dropping the brackets turns the list comprehension into a generator expression. sum pulls the items one at a time.
Exercise 4 🟢 easy explain why
Why does len(x for x in range(5)) fail?
Show solution
A generator doesn't know how many items it will produce until it has produced them all, so it has no length (TypeError: object of type 'generator' has no len()). Use sum(1 for _ in gen) to count, which consumes it, or build a list.
Exercise 5 🟡 medium predict the output
What does this print, in what order?
Show solution
The body runs only between next() calls. C prints when the loop asks for a third value; the function then ends, and its StopIteration ends the loop.
Exercise 6 🟡 medium fix the bug
The second print shows 0. Why? Fix it.
Show solution
The first sum used up the generator. If you need several passes, store the items in a list.
Exercise 7 🟡 medium build it
Write a class Countdown(n) that can be looped over many times, each time yielding n, n-1, …, 1.
Show solution
Because __iter__ returns a new generator each time, the object is a reusable iterable, not a one-shot iterator.
Exercise 8 🟡 medium build it · playground
In playground/read_stream.py, write a generator read_expenses(path). It opens a CSV-like file, skips blank and # lines, and yields (category, amount) tuples. Then compute the total for "food" without building any list.
Show solution
The with block lives inside the generator, so the file is closed when the generator finishes.
Exercise 9 🔴 hard predict the output
What does this print?
Show solution
True then 5. any stops right after it sees 4, so 0 to 4 are used up. The shared iterator continues at 5.
Exercise 10 🔴 hard mini-project
Build a lazy pipeline: numbers() yields 1, 2, 3, … forever. squares(xs) yields the square of each. odd_only(xs) filters out the even values. take(n, xs) yields only the first n items. Print the first 5 odd squares, then show that only 9 numbers were ever generated, by counting inside numbers().
Show solution
Only 9 numbers were ever generated, even though numbers() is infinite. Each stage asks the one before it for the next value only when it needs one. (itertools.islice is the built-in version of take.)