The Library Method: Understanding Generators

Timothy walks into the library with purpose, immediately heading to Margaret's desk.

Timothy: "Margaret, I need the method again. I was working with generators today and something clicked - but then it un-clicked."

Margaret: looks up with interest "Tell me what happened."

Timothy: "I was reading about memory-efficient iteration, and I saw this:"

def count_up(n):
    i = 0
    while i < n:
        yield i
        i += 1

counter = count_up(5)
print(next(counter))  # 0
print(next(counter))  # 1
print(next(counter))  # 2

Output:

0
1
2

Timothy: "It prints 0, then 1, then 2. But here's what confuses me: How does it remember that i was 1? The function returned, right? Where is the state stored between calls?"

Margaret: smiles "That's the perfect question. You're ready to understand one of Python's most elegant features."

Timothy: already pulling out his notebook "Structured English, Pascal, manual Python, then back to the original. Let's do this."

Margaret: "You're getting good at this."

The Confusion

Timothy: runs his complete example

def count_up(n):
    """A simple generator that yields numbers from 0 to n-1."""
    i = 0
    while i < n:
        yield i
        i += 1

# Create a generator object
counter = count_up(5)

# Call next() multiple times
print("Generator Output:")
print(f"First call: {next(counter)}")
print(f"Second call: {next(counter)}")
print(f"Third call: {next(counter)}")
print(f"Fourth call: {next(counter)}")
print(f"Fifth call: {next(counter)}")

# What happens if we call next() again?
try:
    print(f"Sixth call: {next(counter)}")
except StopIteration:
    print("Generator exhausted: StopIteration raised")

Output:

Generator Output:
First call: 0
Second call: 1
Third call: 2
Fourth call: 3
Fifth call: 4
Generator exhausted: StopIteration raised

Timothy: "See? Each call to next() picks up where it left off. The variable i somehow persists between calls. And when it's done, it raises StopIteration. How?"

Margaret: "Let's start by understanding what's really happening here."

The Concept

Margaret: draws on her whiteboard "Before any code, let's think about what a generator is doing."

She writes:

What is a generator?

A generator is a function that can pause and resume
When you call a generator function, it doesn't run immediately - it returns a generator object
Each time you call next(), the function runs until it hits yield
The yield statement pauses execution and saves all local variables
Next time you call next(), it resumes from exactly where it paused
When the function ends naturally, it raises StopIteration

What needs to be saved between calls?

The value of i (and any other local variables)
The current position in the code (which line we're on)
The state of the while loop (are we done?)

Why is this useful?

Memory efficiency: Instead of building a whole list, generate one item at a time
Infinite sequences: Can represent endless streams of data
Lazy evaluation: Only compute values when needed

Timothy: "So it's like the function has a 'pause button' and remembers everything when paused?"

Margaret: "Exactly. Let's build that pause mechanism explicitly in Pascal."

The Blueprint

Margaret: opens the Pascal compiler "In Pascal, we'll create a structure that holds the generator's state, and a function that updates it each time we want the next value."

program GeneratorDemo;

type
  { State structure - holds everything the generator needs to remember }
  TGeneratorState = record
    currentValue: Integer;  { The current value of 'i' }
    maxValue: Integer;      { The limit 'n' }
    exhausted: Boolean;     { Whether we're done }
  end;

{ Initialize a new generator }
function CreateGenerator(n: Integer): TGeneratorState;
var
  state: TGeneratorState;
begin
  state.currentValue := 0;
  state.maxValue := n;
  state.exhausted := False;
  CreateGenerator := state;
end;

{ Get the next value - modifies state and returns current value }
function GetNext(var state: TGeneratorState): Integer;
begin
  if state.exhausted then
  begin
    GetNext := -1;  { Signal exhaustion }
    Exit;
  end;

  { Return current value }
  GetNext := state.currentValue;

  { Update state for next call }
  state.currentValue := state.currentValue + 1;

  { Check if we're done }
  if state.currentValue >= state.maxValue then
    state.exhausted := True;
end;

var
  counter: TGeneratorState;
  result: Integer;

begin
  WriteLn('Generator Output:');

  { Create generator state }
  counter := CreateGenerator(5);

  { Call GetNext multiple times }
  WriteLn('First call: ', GetNext(counter));
  WriteLn('Second call: ', GetNext(counter));
  WriteLn('Third call: ', GetNext(counter));
  WriteLn('Fourth call: ', GetNext(counter));
  WriteLn('Fifth call: ', GetNext(counter));

  { Try calling after exhaustion - Pascal doesn't have exceptions }
  result := GetNext(counter);
  if result = -1 then
    WriteLn('Generator exhausted: StopIteration raised');
end.

Margaret compiles and runs it

Output:

Generator Output:
First call: 0
Second call: 1
Third call: 2
Fourth call: 3
Fifth call: 4
Generator exhausted: StopIteration raised

Timothy: studies the code "So the TGeneratorState record is storing everything - the current value, the limit, whether we're done..."

Margaret: "Exactly. Each call to GetNext() reads the state, returns the current value, then updates the state for next time. That's all a generator does - Python just hides the state management."

Timothy: "And Pascal doesn't have exceptions like Python, so you check the return value instead?"

Margaret: "Right. Different mechanism, same concept - signaling when we're done."

Timothy: "Let me write the manual Python version. I think I see how to do this with a class."

The Manual

Timothy: types with growing confidence

class ManualGenerator:
    """Explicitly manages generator state like our Pascal version."""

    def __init__(self, n):
        # Initialize state
        self.current_value = 0
        self.max_value = n
        self.exhausted = False

    def get_next(self):
        """Returns next value and updates state."""
        # Check if exhausted
        if self.exhausted:
            raise StopIteration("Generator exhausted: StopIteration raised")

        # Get current value to return
        result = self.current_value

        # Update state for next call
        self.current_value += 1

        # Check if we'll be exhausted after this
        if self.current_value >= self.max_value:
            self.exhausted = True

        return result

# Test the manual generator
print("Generator Output:")

counter = ManualGenerator(5)

print(f"First call: {counter.get_next()}")
print(f"Second call: {counter.get_next()}")
print(f"Third call: {counter.get_next()}")
print(f"Fourth call: {counter.get_next()}")
print(f"Fifth call: {counter.get_next()}")

# Try calling after exhaustion
try:
    counter.get_next()
except StopIteration as e:
    print(e)

Timothy runs it

Output:

Generator Output:
First call: 0
Second call: 1
Third call: 2
Fourth call: 3
Fifth call: 4
Generator exhausted: StopIteration raised

Timothy: triumphant "Same output! The class instance holds the state between calls, just like the Pascal record."

Margaret: "Perfect. You've built what Python automates with yield. Now look at the original code again."

The Revelation

Margaret brings up the original generator code

def count_up(n):
    i = 0              # ← This becomes state.current_value
    while i < n:       # ← This becomes the exhausted check
        yield i        # ← This saves state, returns value, and pauses
        i += 1         # ← This happens when we resume
    # When function ends ← This raises StopIteration

# When you call count_up(5), Python creates an object that:
# - Stores all local variables (i, n)
# - Remembers the current line of execution
# - Can pause at yield and resume later

counter = count_up(5)

# Each next() call:
# 1. Resumes from where yield paused
# 2. Runs until next yield
# 3. Returns the yielded value
# 4. Saves all state again

print("Generator Output:")
print(f"First call: {next(counter)}")   # Runs to yield, returns 0, saves i=1
print(f"Second call: {next(counter)}")  # Resumes with i=1, yields 1, saves i=2
print(f"Third call: {next(counter)}")   # Resumes with i=2, yields 2, saves i=3
print(f"Fourth call: {next(counter)}")  # Resumes with i=3, yields 3, saves i=4
print(f"Fifth call: {next(counter)}")   # Resumes with i=4, yields 4, saves i=5

# Try to get one more
try:
    next(counter)
except StopIteration:
    print("Generator exhausted: StopIteration raised")

Output:

Generator Output:
First call: 0
Second call: 1
Third call: 2
Fourth call: 3
Fifth call: 4
Generator exhausted: StopIteration raised

Timothy: "So yield is doing everything our manual class did - storing state, checking if done, updating variables..."

Margaret: "Exactly. The generator object is like your ManualGenerator class. Python creates it automatically when you call a function with yield."

Timothy: "And i persists because it's stored in the generator object's state."

Margaret: "Right. Just like self.current_value in your class, or state.currentValue in Pascal."

Timothy: leans back "It's not magic. It's just automatic state management."

The Principle

Margaret: closes her notebook "So when should you use generators?"

Timothy: thinking "When you need to remember state between function calls?"

Margaret: "That's part of it. More specifically:"

Use generators when:

Processing large datasets that won't fit in memory
Creating infinite sequences (like count_up(float('inf')))
Lazy evaluation - only compute values when needed
You need to pause expensive operations and resume later
Pipeline processing (one generator feeding another)

Don't use generators when:

You need random access to items (generators are forward-only)
You need the full list multiple times (generator exhausts after one pass)
The overhead of state management exceeds the memory savings

Timothy: "So that's why generators are 'memory efficient' - you're not building a whole list."

Margaret: "Exactly. Compare these two:"

# List: All values in memory at once
numbers = [i for i in range(1000000)]  # Uses lots of memory

# Generator: One value at a time
numbers = (i for i in range(1000000))  # Minimal memory

Timothy: "The generator version only creates values as you ask for them."

Margaret: "Right. And now you understand how it does that - by maintaining state between calls, just like we built manually."

Summary

🎯 What We Proved:
All three implementations produced identical output:

Sequence: 0, 1, 2, 3, 4
Behavior: StopIteration when exhausted
State persistence: Each call remembers previous values

💡 What We Learned:

Generators are functions that can pause and resume
yield saves all local variables and the execution position
The generator object stores this state between next() calls
It's automatic state management, not magic
State can be explicitly managed with classes (what we built manually)

🔧 The Library Method:

Structured English - Generators pause, save state, and resume
Pascal Blueprint - Explicit state record with update function
Manual Python - Class-based state management
Pythonic Revelation - yield automates everything we built

Key Insight: A generator is just a function with automatic state persistence. Python creates an object that holds your local variables and remembers where you paused - exactly what we built manually in Pascal and Python.

Next in The Library Method: Understanding Context Managers - What does the with statement actually do?

Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.

Search This Blog

Tech-Reader.blog