The Secret Life of Python: Using imap for Streaming Results
The Secret Life of Python: Using imap for Streaming Results How to process data as it finishes, not when everything is done #Python #Multiprocessing #StreamingResults #imap Margaret is a senior software engineer. Timothy is her junior colleague. They work in a grand Victorian library in London — the kind of place where code quality is the unspoken objective, and craftsmanship is the only thing that matters. Episode 38 Timothy was happy with his new Process Pool —a tool that let him hire a "fleet" of workers to analyze chess matches across all the cores of his CPU. But he noticed a frustrating bottleneck. Some of his chess matches were "Blitz" games (taking seconds to analyze), while others were "Marathons" (taking minutes). "Margaret," Timothy said, "the standard pool.map command is making me wait. If the first match in the pile is a slow Marathon, I don't see the results of the nine fast Blitz matches until that one slow on...