The Secret Life of Python: The Iterator Protocol 1 - Why For Loops Are Magic

 

The Secret Life of Python: The Iterator Protocol 1 - Why For Loops Are Magic





Timothy was explaining Python to a colleague from C++ when he got stumped. "So in Python, you can loop over lists, dictionaries, files, strings, ranges, sets... how does for know what to do with all these different types?"

Margaret overheard and smiled. "That's the iterator protocol - one of Python's most elegant designs. Every type that works with for speaks the same language, and you can teach your own objects that language too. Let me show you the magic."

The Puzzle: For Loops Work on Everything

Timothy showed Margaret what confused him:

def demonstrate_for_loop_versatility():
    """For loops work on so many different types!"""

    # Loop over a list
    for item in [1, 2, 3]:
        print(item, end=' ')
    print("← list")

    # Loop over a string
    for char in "hello":
        print(char, end=' ')
    print("← string")

    # Loop over a dictionary
    for key in {'a': 1, 'b': 2}:
        print(key, end=' ')
    print("← dict keys")

    # Loop over a file
    with open('example.txt', 'w') as f:
        f.write('line1\nline2\nline3')

    with open('example.txt') as f:
        for line in f:
            print(line.strip(), end=' ')
    print("← file lines")

    # Loop over a range
    for num in range(3):
        print(num, end=' ')
    print("← range")

    # Loop over a set
    for item in {10, 20, 30}:
        print(item, end=' ')
    print("← set")

demonstrate_for_loop_versatility()

Output:

1 2 3 ← list
h e l l o ← string
a b ← dict keys
line1 line2 line3 ← file lines
0 1 2 ← range
10 20 30 ← set

"See?" Timothy pointed. "The for loop syntax is identical for all these different types. How does Python know how to iterate over each one?"

The Iterator Protocol: Python's Iteration Contract

Margaret sketched out the concept:

"""
The Iterator Protocol: Two simple methods that make iteration work

ITERABLE: Any object with __iter__() method
- Returns an iterator

ITERATOR: Any object with __next__() method
- Returns next item
- Raises StopIteration when done
- Should also have __iter__() that returns self

THE FOR LOOP CONTRACT:
When Python sees:    for item in obj:
It actually does:    
    iterator = iter(obj)      # Calls obj.__iter__()
    while True:
        try:
            item = next(iterator)  # Calls iterator.__next__()
            # loop body
        except StopIteration:
            break

Every object that works with 'for' implements this protocol!
"""

def demonstrate_for_loop_expansion():
    """Show what a for loop really does"""

    items = [1, 2, 3]

    print("Using for loop:")
    for item in items:
        print(f"  {item}")

    print("\nWhat Python actually does:")
    iterator = iter(items)  # Get iterator
    while True:
        try:
            item = next(iterator)  # Get next item
            print(f"  {item}")
        except StopIteration:  # No more items
            break

    print("\n✓ Same result!")

demonstrate_for_loop_expansion()

Output:

Using for loop:
  1
  2
  3

What Python actually does:
  1
  2
  3

✓ Same result!

Building a Simple Iterator from Scratch

"Let me show you how to create your own iterator," Margaret said.

class CountDown:
    """Simple iterator that counts down from n to 1"""

    def __init__(self, start):
        self.current = start

    def __iter__(self):
        """Return the iterator object (self)"""
        return self

    def __next__(self):
        """Return the next value or raise StopIteration"""
        if self.current <= 0:
            raise StopIteration

        value = self.current
        self.current -= 1
        return value

def demonstrate_custom_iterator():
    """Use our custom iterator"""

    print("Counting down:")
    for num in CountDown(5):
        print(num, end=' ')
    print("\n")

    # Show it works with manual iteration too
    print("Manual iteration:")
    countdown = CountDown(3)
    print(f"  next(): {next(countdown)}")
    print(f"  next(): {next(countdown)}")
    print(f"  next(): {next(countdown)}")

    try:
        next(countdown)  # Should raise StopIteration
    except StopIteration:
        print("  StopIteration raised - done!")

demonstrate_custom_iterator()

Output:

Counting down:
5 4 3 2 1 

Manual iteration:
  next(): 3
  next(): 2
  next(): 1
  StopIteration raised - done!

Timothy studied the code. "So the iterator remembers where it is - the current value - and each time I call next(), it advances and returns the next value."

"Exactly," Margaret confirmed. "But there's a limitation with this design. Watch what happens if you try to iterate twice."

She typed quickly:

countdown = CountDown(3)

print("First loop:")
for num in countdown:
    print(num, end=' ')

print("\n\nSecond loop:")
for num in countdown:
    print(num, end=' ')  # Will this work?

Output:

First loop:
3 2 1 

Second loop:

"Nothing on the second loop!" Timothy exclaimed. "Why?"

"Because the iterator exhausted itself on the first loop. The current value is now 0, and it stays 0. If you want to iterate again, you need a fresh iterator. This is where we need to separate the concepts of iterable and iterator."

Separating Iterable from Iterator

Margaret explained an important distinction:

"""
BEST PRACTICE: Separate iterable and iterator

Iterable: Container that can be iterated over
Iterator: The actual object that does the iterating

This allows:
- Multiple simultaneous iterations
- Reusing the iterable
- Cleaner design
"""

class CountDown:
    """Iterable that creates countdown iterators"""

    def __init__(self, start):
        self.start = start

    def __iter__(self):
        """Return a NEW iterator each time"""
        return CountDownIterator(self.start)

class CountDownIterator:
    """The actual iterator"""

    def __init__(self, start):
        self.current = start

    def __iter__(self):
        """Iterators should return themselves"""
        return self

    def __next__(self):
        """Return next value"""
        if self.current <= 0:
            raise StopIteration

        value = self.current
        self.current -= 1
        return value

def demonstrate_multiple_iterations():
    """Show why separating iterable/iterator matters"""

    countdown = CountDown(3)

    print("First iteration:")
    for num in countdown:
        print(num, end=' ')
    print()

    print("\nSecond iteration (works because we get a fresh iterator!):")
    for num in countdown:
        print(num, end=' ')
    print()

    print("\nMultiple simultaneous iterations:")
    iter1 = iter(countdown)
    iter2 = iter(countdown)

    print(f"  iter1: {next(iter1)}, {next(iter1)}")
    print(f"  iter2: {next(iter2)}, {next(iter2)}")
    print("  ✓ Independent iterators!")

demonstrate_multiple_iterations()

Output:

First iteration:
3 2 1

Second iteration (works because we get a fresh iterator!):
3 2 1

Multiple simultaneous iterations:
  iter1: 3, 2
  iter2: 3, 2
  ✓ Independent iterators!

Built-in Iterables Explained

Timothy wanted to understand how built-in types work:

def explore_builtin_iterables():
    """Understand how built-in types implement iteration"""

    # Lists
    my_list = [1, 2, 3]
    print("List:")
    print(f"  Has __iter__: {hasattr(my_list, '__iter__')}")
    print(f"  iter() returns: {type(iter(my_list))}")

    # Strings
    my_string = "hello"
    print("\nString:")
    print(f"  Has __iter__: {hasattr(my_string, '__iter__')}")
    print(f"  iter() returns: {type(iter(my_string))}")

    # Dictionaries
    my_dict = {'a': 1, 'b': 2}
    print("\nDict:")
    print(f"  Has __iter__: {hasattr(my_dict, '__iter__')}")
    print(f"  iter() returns: {type(iter(my_dict))}")
    print(f"  Default iteration: {list(my_dict)}")  # Keys
    print(f"  Values: {list(my_dict.values())}")
    print(f"  Items: {list(my_dict.items())}")

    # Files
    with open('temp.txt', 'w') as f:
        f.write('line1\nline2')

    with open('temp.txt') as f:
        print("\nFile:")
        print(f"  Has __iter__: {hasattr(f, '__iter__')}")
        print(f"  Iterates over: lines")

explore_builtin_iterables()

Real-World Use Case 1: Custom Collection

Margaret showed a practical example:

class Playlist:
    """Music playlist that can be iterated over"""

    def __init__(self, name):
        self.name = name
        self.songs = []

    def add_song(self, song):
        self.songs.append(song)

    def __iter__(self):
        """Return iterator over songs"""
        return iter(self.songs)  # Delegate to list's iterator

    def __len__(self):
        return len(self.songs)

def demonstrate_custom_collection():
    """Show custom iterable collection"""

    playlist = Playlist("Favorites")
    playlist.add_song("Song A")
    playlist.add_song("Song B")
    playlist.add_song("Song C")

    print(f"Playlist: {playlist.name}")
    print(f"  Songs ({len(playlist)}):")
    for song in playlist:
        print(f"    - {song}")

    # Works with all iteration tools
    print(f"\n  First song: {next(iter(playlist))}")
    print(f"  All songs: {list(playlist)}")

demonstrate_custom_collection()

Output:

Playlist: Favorites
  Songs (3):
    - Song A
    - Song B
    - Song C

  First song: Song A
  All songs: ['Song A', 'Song B', 'Song C']

Real-World Use Case 2: Pagination

class PaginatedAPI:
    """Iterator for paginated API results"""

    def __init__(self, page_size=10, total_items=100):
        self.page_size = page_size
        self.total_items = total_items
        self.current_item = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.current_item >= self.total_items:
            raise StopIteration

        # Simulate fetching a page of results
        page_start = self.current_item
        page_end = min(self.current_item + self.page_size, self.total_items)

        items = list(range(page_start, page_end))
        self.current_item = page_end

        return items

def demonstrate_pagination():
    """Show pagination iterator"""

    print("Fetching results in pages:")
    api = PaginatedAPI(page_size=25, total_items=87)

    for page_num, page in enumerate(api, 1):
        print(f"  Page {page_num}: {len(page)} items (first: {page[0]}, last: {page[-1]})")

    print(f"\n✓ Fetched all items without loading everything into memory!")

demonstrate_pagination()

Output:

Fetching results in pages:
  Page 1: 25 items (first: 0, last: 24)
  Page 2: 25 items (first: 25, last: 49)
  Page 3: 25 items (first: 50, last: 74)
  Page 4: 12 items (first: 75, last: 86)

✓ Fetched all items without loading everything into memory!

Real-World Use Case 3: Infinite Sequences

Timothy was intrigued: "Can iterators be infinite?"

class FibonacciIterator:
    """Infinite Fibonacci sequence"""

    def __init__(self):
        self.a, self.b = 0, 1

    def __iter__(self):
        return self

    def __next__(self):
        value = self.a
        self.a, self.b = self.b, self.a + self.b
        return value

class Counter:
    """Infinite counter starting from n"""

    def __init__(self, start=0):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        value = self.current
        self.current += 1
        return value

def demonstrate_infinite_iterators():
    """Show infinite sequences (safely!)"""

    print("First 10 Fibonacci numbers:")
    fib = FibonacciIterator()
    for i, num in enumerate(fib):
        print(num, end=' ')
        if i >= 9:  # Stop after 10
            break
    print()

    print("\nCounter starting from 100 (first 5):")
    counter = Counter(100)
    for i, num in enumerate(counter):
        print(num, end=' ')
        if i >= 4:
            break
    print()

    print("\nđź’ˇ Infinite iterators + break = controlled infinite sequences!")

demonstrate_infinite_iterators()

Output:

First 10 Fibonacci numbers:
0 1 1 2 3 5 8 13 21 34 

Counter starting from 100 (first 5):
100 101 102 103 104 

đź’ˇ Infinite iterators + break = controlled infinite sequences!

The iter() Function's Two Forms

Margaret showed a lesser-known feature:

def demonstrate_iter_with_sentinel():
    """iter() has a two-argument form!"""

    # Form 1: Normal iterator
    # iter(iterable) → iterator

    # Form 2: Callable with sentinel
    # iter(callable, sentinel) → iterator
    # Calls callable repeatedly until it returns sentinel

    import random
    random.seed(42)

    def roll_die():
        """Simulate rolling a die"""
        return random.randint(1, 6)

    print("Rolling die until we get a 6:")
    # iter(callable, sentinel) - calls roll_die() until it returns 6
    for roll in iter(roll_die, 6):
        print(f"  Rolled: {roll}")
    print("  Got 6! Stopped.\n")

    # Another example: Reading file blocks
    def read_block():
        """Simulate reading blocks from a file"""
        blocks = [b'data1', b'data2', b'', b'data3']
        return blocks.pop(0) if blocks else b''

    print("Reading blocks until empty:")
    for block in iter(read_block, b''):
        print(f"  Block: {block}")
    print("  Empty block! Stopped.")

demonstrate_iter_with_sentinel()

Iterator Exhaustion

Timothy learned an important gotcha:

def demonstrate_iterator_exhaustion():
    """Iterators can only be used once"""

    my_list = [1, 2, 3]

    # Lists are iterables (can create multiple iterators)
    print("List (iterable):")
    print(f"  First loop: {list(my_list)}")
    print(f"  Second loop: {list(my_list)}")
    print("  ✓ Works multiple times!\n")

    # Iterators exhaust
    my_iterator = iter([1, 2, 3])

    print("Iterator:")
    print(f"  First loop: {list(my_iterator)}")
    print(f"  Second loop: {list(my_iterator)}")  # Empty!
    print("  ✗ Iterator exhausted after first use!\n")

    # Checking if exhausted
    my_iterator = iter([1, 2, 3])
    print("Checking exhaustion:")
    print(f"  Has items: {next(my_iterator, 'EMPTY') != 'EMPTY'}")
    print(f"  Next item: {next(my_iterator)}")
    print(f"  Next item: {next(my_iterator)}")
    print(f"  Next item: {next(my_iterator, 'EMPTY')}")  # Exhausted

demonstrate_iterator_exhaustion()

Output:

List (iterable):
  First loop: [1, 2, 3]
  Second loop: [1, 2, 3]
  ✓ Works multiple times!

Iterator:
  First loop: [1, 2, 3]
  Second loop: []
  ✗ Iterator exhausted after first use!

Checking exhaustion:
  Has items: True
  Next item: 2
  Next item: 3
  Next item: EMPTY

Built-in Iterator Tools

Margaret showed Python's powerful iterator utilities:

import itertools

def demonstrate_iterator_tools():
    """Python's built-in iterator tools"""

    # itertools.count - infinite counter
    print("itertools.count (first 5):")
    for i, num in enumerate(itertools.count(10, 2)):
        print(num, end=' ')
        if i >= 4:
            break
    print()

    # itertools.cycle - repeat sequence infinitely
    print("\nitertools.cycle (first 10):")
    for i, item in enumerate(itertools.cycle(['A', 'B', 'C'])):
        print(item, end=' ')
        if i >= 9:
            break
    print()

    # itertools.chain - combine iterables
    print("\nitertools.chain:")
    combined = itertools.chain([1, 2], ['a', 'b'], [10, 20])
    print(f"  {list(combined)}")

    # itertools.islice - slice an iterator
    print("\nitertools.islice (items 2-5 from infinite counter):")
    print(f"  {list(itertools.islice(itertools.count(), 2, 6))}")

    # zip - iterate multiple sequences together
    print("\nzip:")
    names = ['Alice', 'Bob', 'Charlie']
    ages = [25, 30, 35]
    for name, age in zip(names, ages):
        print(f"  {name}: {age}")

    # enumerate - add indices
    print("\nenumerate:")
    for i, fruit in enumerate(['apple', 'banana', 'cherry'], start=1):
        print(f"  {i}. {fruit}")

demonstrate_iterator_tools()

Output:

itertools.count (first 5):
10 12 14 16 18 

itertools.cycle (first 10):
A B C A B C A B C A 

itertools.chain:
  [1, 2, 'a', 'b', 10, 20]

itertools.islice (items 2-5 from infinite counter):
  [2, 3, 4, 5]

zip:
  Alice: 25
  Bob: 30
  Charlie: 35

enumerate:
  1. apple
  2. banana
  3. cherry

Iterator vs Iterable: The Key Difference

def demonstrate_iterator_vs_iterable():
    """Understand the crucial difference"""

    # List is ITERABLE (not iterator)
    my_list = [1, 2, 3]
    print("List (iterable):")
    print(f"  Has __iter__: {hasattr(my_list, '__iter__')}")
    print(f"  Has __next__: {hasattr(my_list, '__next__')}")
    print(f"  Is its own iterator: {iter(my_list) is my_list}")

    # Get iterator from iterable
    my_iterator = iter(my_list)
    print("\nIterator from list:")
    print(f"  Has __iter__: {hasattr(my_iterator, '__iter__')}")
    print(f"  Has __next__: {hasattr(my_iterator, '__next__')}")
    print(f"  Is its own iterator: {iter(my_iterator) is my_iterator}")

    print("\nKEY DIFFERENCE:")
    print("  Iterable: Can create iterators (__iter__)")
    print("  Iterator: Does the iteration (__next__)")
    print("  Iterator is also iterable (returns self from __iter__)")

demonstrate_iterator_vs_iterable()

Output:

List (iterable):
  Has __iter__: True
  Has __next__: False
  Is its own iterator: False

Iterator from list:
  Has __iter__: True
  Has __next__: True
  Is its own iterator: True

KEY DIFFERENCE:
  Iterable: Can create iterators (__iter__)
  Iterator: Does the iteration (__next__)
  Iterator is also iterable (returns self from __iter__)

The Pythonic Shortcut: A Glimpse at Generators

Timothy looked at all the iterator code they'd written. "This is powerful, but writing __iter__ and __next__ and tracking state with self.current - it feels like a lot of boilerplate for something so common."

"You're absolutely right," Margaret said with a knowing smile. "And Python's designers thought the same thing. That's why they created generators - a shortcut for creating iterators using the yield keyword."

"Yield?" Timothy asked.

"Watch this." Margaret opened a new window and typed:

def countdown_generator(n):
    """Generator version - much simpler!"""
    while n > 0:
        yield n
        n -= 1

# Use it exactly like our iterator
for num in countdown_generator(5):
    print(num, end=' ')
print()

# Works multiple times
for num in countdown_generator(3):
    print(num, end=' ')

Output:

5 4 3 2 1 
3 2 1 

Timothy stared. "That's it? Just yield instead of all that __iter__ and __next__ code?"

"That's it," Margaret confirmed. "When you use yield, Python automatically creates an iterator for you. It handles __iter____next__, state preservation, and raising StopIteration. Everything we just learned about iterators - Python does it for you."

"So why did we learn all that iterator protocol stuff if generators are easier?"

Margaret leaned back. "Because understanding the iterator protocol shows you how Python iteration actually works. Generators are magical-looking until you realize they're just syntactic sugar for the iterator protocol. Now you understand both the mechanism and the convenience."

She pulled up a comparison:

# Iterator class - explicit protocol
class CountDown:
    def __init__(self, start):
        self.start = start

    def __iter__(self):
        return CountDownIterator(self.start)

class CountDownIterator:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

# Generator - protocol handled automatically
def countdown_generator(start):
    while start > 0:
        yield start
        start -= 1

# Both work identically
print("Iterator:", list(CountDown(3)))
print("Generator:", list(countdown_generator(3)))

Output:

Iterator: [3, 2, 1]
Generator: [3, 2, 1]

"They produce the exact same result," Timothy observed. "The generator is just... cleaner."

"Much cleaner. And that's why generators are the Pythonic way to create custom iterators in practice. But now you understand why they work - they're implementing the iterator protocol under the hood."

Margaret stood up. "In our next conversation, we'll dive deep into generators - how they work, why they're memory-efficient, and all the powerful things you can do with them. But first, let me show you some common mistakes to avoid with iterators."

Common Pitfalls

Timothy wanted to know what to watch out for:

def pitfall_1_modifying_while_iterating():
    """Pitfall: Modifying a list while iterating"""

    # ❌ WRONG - Modifying during iteration
    items = [1, 2, 3, 4, 5]
    print("Attempting to remove even numbers:")
    try:
        for item in items:
            if item % 2 == 0:
                items.remove(item)  # Modifies during iteration! Can skip items or raise RuntimeError
        print(f"  Result: {items}")
        print("  ✗ Missed item 4! (Iterator got confused)")
    except RuntimeError as e:
        print(f"  ✗ RuntimeError: {e}")

    # ✓ CORRECT - Iterate over copy
    items = [1, 2, 3, 4, 5]
    print("\nIterating over copy:")
    for item in items[:]:  # Slice creates copy
        if item % 2 == 0:
            items.remove(item)
    print(f"  Result: {items}")
    print("  ✓ Correct!")

    # ✓ CORRECT - List comprehension
    items = [1, 2, 3, 4, 5]
    print("\nList comprehension:")
    items = [item for item in items if item % 2 != 0]
    print(f"  Result: {items}")
    print("  ✓ Best approach!")

def pitfall_2_iterator_consumed():
    """Pitfall: Using iterator twice"""

    iterator = iter([1, 2, 3])

    print("\nFirst consumption:")
    result1 = list(iterator)
    print(f"  {result1}")

    print("\nSecond consumption:")
    result2 = list(iterator)
    print(f"  {result2}")
    print("  ✗ Iterator was already exhausted!")

def pitfall_3_infinite_without_break():
    """Pitfall: Infinite iterator without break"""

    print("\nInfinite iterator needs break:")
    print("  for i in itertools.count():")
    print("      if i > 5: break  # ← Must have stopping condition!")

pitfall_1_modifying_while_iterating()
pitfall_2_iterator_consumed()
pitfall_3_infinite_without_break()

Testing Iterator Behavior

Margaret showed testing patterns:

import pytest

def test_custom_iterator():
    """Test a custom iterator"""

    class SimpleIterator:
        def __init__(self, data):
            self.data = data
            self.index = 0

        def __iter__(self):
            return self

        def __next__(self):
            if self.index >= len(self.data):
                raise StopIteration
            value = self.data[self.index]
            self.index += 1
            return value

    iterator = SimpleIterator([1, 2, 3])

    # Test iteration
    assert next(iterator) == 1
    assert next(iterator) == 2
    assert next(iterator) == 3

    # Test StopIteration
    with pytest.raises(StopIteration):
        next(iterator)

def test_iterable_reusable():
    """Test that iterable can be iterated multiple times"""

    class Reusable:
        def __init__(self, data):
            self.data = data

        def __iter__(self):
            return iter(self.data)

    obj = Reusable([1, 2, 3])

    # First iteration
    result1 = list(obj)
    assert result1 == [1, 2, 3]

    # Second iteration (should work!)
    result2 = list(obj)
    assert result2 == [1, 2, 3]

def test_iterator_exhaustion():
    """Test that iterator exhausts"""

    iterator = iter([1, 2, 3])

    # Exhaust iterator
    list(iterator)

    # Should be empty now
    assert list(iterator) == []

# Run with: pytest test_iterators.py -v

The Library Metaphor

Margaret brought it back to the library:

"Think of iteration like checking out books from a catalog," she said.

"The catalog itself (an iterable) isn't the process of checking out books - it's the thing that enables checking out. When you start checking out books, you get a bookmark (an iterator) that tracks your progress through the catalog.

"The bookmark has two jobs:

  1. Return the next book in the catalog
  2. Remember where you are

"You can have multiple bookmarks in the same catalog (multiple iterators from one iterable), and each tracks its position independently. Once a bookmark reaches the end, it can't be reused - you need a new bookmark to go through the catalog again.

"This is why for loops work with so many types. Every type implements the same 'catalog and bookmark' protocol: provide a bookmark (__iter__), and the bookmark knows how to move forward (__next__)."

Key Takeaways

Margaret summarized:

"""
ITERATOR PROTOCOL KEY TAKEAWAYS:

1. Two key methods:
   - __iter__(): Returns an iterator
   - __next__(): Returns next item or raises StopIteration

2. Iterable vs Iterator:
   - Iterable: Can create iterators (has __iter__)
   - Iterator: Does the iterating (has __next__ and __iter__)
   - Iterator's __iter__ should return self

3. For loops use this protocol:
   - for item in obj: → iter(obj) then next() repeatedly
   - Works with any object implementing the protocol

4. Separation of concerns:
   - Iterable: The container/sequence
   - Iterator: The state of iteration
   - Allows multiple simultaneous iterations

5. Iterator exhaustion:
   - Iterators can only be used once
   - Iterables can create fresh iterators
   - Store iterables, not iterators

6. Generators: The Pythonic shortcut:
   - Use 'yield' instead of __iter__ and __next__
   - Python handles the protocol automatically
   - Cleaner, more readable code
   - Next article will explore generators deeply

7. Real-world uses:
   - Custom collections
   - Pagination (lazy loading)
   - Infinite sequences
   - Streaming data
   - File processing

8. Built-in tools:
   - itertools: count, cycle, chain, islice, etc.
   - zip: combine sequences
   - enumerate: add indices
   - iter(callable, sentinel): advanced form

9. Common pitfalls:
   - Modifying while iterating (can skip items or raise RuntimeError)
   - Reusing exhausted iterators
   - Infinite iterators without break
   - Confusing iterable with iterator

10. Benefits:
    - Memory efficient (one item at a time)
    - Lazy evaluation (compute on demand)
    - Uniform interface for different types
    - Enables powerful composition

11. When to use:
    - Custom collections
    - Large datasets (don't load all at once)
    - Infinite sequences
    - API pagination
    - File/stream processing
"""

Timothy nodded, understanding. "So the iterator protocol is why for is so versatile. Every type speaks the same language - __iter__ to start, __next__ to continue, StopIteration to finish. And I can make my own types speak this language too!"

"Exactly," Margaret said. "The iterator protocol is one of Python's most elegant designs. Two simple methods, and suddenly any object can work with for loops, list comprehensions, next()zip(), and all the iterator tools."

"And generators," Timothy added, "are just the convenient way to implement this protocol without all the boilerplate."

"Right. Which is why our next conversation will be all about generators - how they work under the hood, why they're so memory-efficient, and all the powerful patterns you can build with them. But now you understand the foundation: the iterator protocol that makes it all possible."

With that knowledge, Timothy could create custom iterables, understand how iteration really works, recognize the protocol in action throughout Python, and appreciate why generators are such a powerful feature - because they automate the protocol he now understood deeply.


Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.

Comments

Popular posts from this blog

The New ChatGPT Reason Feature: What It Is and Why You Should Use It

Insight: The Great Minimal OS Showdown—DietPi vs Raspberry Pi OS Lite

Raspberry Pi Connect vs. RealVNC: A Comprehensive Comparison