The Secret Life of Python: Using `imap` for Streaming Results

How to process data as it finishes, not when everything is done

#Python #Multiprocessing #StreamingResults #imap

Margaret is a senior software engineer. Timothy is her junior colleague. They work in a grand Victorian library in London — the kind of place where code quality is the unspoken objective, and craftsmanship is the only thing that matters.

Episode 38

Timothy was happy with his new Process Pool—a tool that let him hire a "fleet" of workers to analyze chess matches across all the cores of his CPU. But he noticed a frustrating bottleneck.

Some of his chess matches were "Blitz" games (taking seconds to analyze), while others were "Marathons" (taking minutes).

"Margaret," Timothy said, "the standard pool.map command is making me wait. If the first match in the pile is a slow Marathon, I don't see the results of the nine fast Blitz matches until that one slow one finishes. I’m sitting here with an empty screen when I could be working on the results that are already done!"

Margaret nodded. "You’re dealing with Blocking. The standard map command is a perfectionist; it waits until every single worker has finished the entire stack of work before it hands you the final list. If you want to be an Impatient Manager, you need a stream, not a pile."

The Streaming Concept

"Think of it like a Conveyor Belt," Margaret explained. "In the standard 'perfectionist' way, the belt doesn't move until the very last box is packed. But if you use pool.imap, the belt starts moving the second the first worker is done. You can pick up the fast results as they arrive, even while the slow worker is still sweating over their desk."

Seeing the Results in Real-Time

Timothy updated his code. He traded the static "pile" of results for a dynamic loop that watched the belt.

from multiprocessing import Pool
import time
import random

def analyze_match(match_id):
    # Some matches are fast (0.5s), some are slow (3s)
    duration = random.uniform(0.5, 3.0)
    time.sleep(duration) 
    return f"Match #{match_id} analyzed in {duration:.1f}s"

if __name__ == "__main__":
    match_ids = range(1, 11) # A stack of 10 matches
    
    # We open a Pool with 4 workers
    with Pool(processes=4) as pool:
        print("--- The Impatient Manager is watching the belt ---")
        
        # 'imap' returns an iterator—a live stream of results
        result_stream = pool.imap(analyze_match, match_ids)

        for result in result_stream:
            # This prints the MOMENT each task is ready
            print(f"Manager received: {result}")

Timothy watched the screen. Instead of a long, silent pause followed by ten lines appearing at once, the results started "popping" onto the screen one by one.

"I'm still getting them in the order I sent them," Timothy noted, "but I'm not waiting for the entire list to be ready. It feels much more alive."

The Specialist’s Agility: Lazy Evaluation

"That's the power of Lazy Evaluation," Margaret said. "You aren't loading the results into a massive, heavy pile in memory. You’re letting them flow through your program. If you had a million matches, the 'perfectionist' map might crash your computer by trying to build a giant million-item list before showing you anything. imap just keeps the stream moving."

Timothy looked at the steady drip of results. He had moved from being a manager who waits for a final report to a manager who watches the live dashboard.

Margaret’s Cheat Sheet: The Streaming Pool

Choosing Your Strategy

map()
Blocking
When to use: Small batches where you need a complete list at once.

imap()
Streaming
When to use: Large data where you want to see results as they finish.

imap_unordered()
Fastest
When to use: When you want results the instant they are done, regardless of order.

The Specialist's Wisdom

Lazy Evaluation: imap is "lazy" (in a good way). It only processes and yields results as you ask for them, making it incredibly memory-efficient for Big Data.
Order Preservation: Standard imap still respects your input order. If Item #1 is slow, the stream will wait for it before showing Items #2 and #3.
The Chunksize Tip: For millions of tiny tasks, use pool.imap(func, data, chunksize=100). This sends work to your CPU in "batches" instead of one by one, which is much more efficient.

Aaron Rose is a software engineer and technology writer at tech-reader.blog.

Catch up on the latest explainer videos, podcasts, and industry discussions below.

Search This Blog

Tech-Reader.blog