The Secret Life of Python: How to Use All Your CPU Cores

A practical guide to multiprocessing for CPU-bound tasks

#Python #Coding #Multiprocessing #Threads

Margaret is a senior software engineer. Timothy is her junior colleague. They work in a grand Victorian library in London — the kind of place where code quality is the unspoken objective, and craftsmanship is the only thing that matters.

Episode 36

Timothy was ready to break the "Glass Ceiling." He knew now that his "Grandmaster Analysis" engine was too heavy for threads. It needed raw power, not just a guest waiting for a turn to speak.

"To use all your cores," Margaret said, pointing to his laptop, "we have to stop trying to share one copy of Python. We’re going to give every worker their own room, their own memory, and—most importantly—their own Global Interpreter Lock."

Timothy opened the multiprocessing module. It looked surprisingly similar to the threading module, but Margaret warned him: "The code looks the same, but under the hood, a process is a much heavier beast than a thread."

Moving to Parallel Worlds

Timothy wrote a heavy calculation function—something to simulate calculating millions of chess positions—and deployed it across four separate processes.

import multiprocessing
import time

def heavy_analysis(name):
    print(f"Process {name}: Starting deep analysis...")
    # Simulating a heavy CPU task (sum of squares)
    sum(i * i for i in range(10_000_000))
    print(f"Process {name}: Analysis complete.")

if __name__ == "__main__":
    processes = []
    
    # Check how many stages we can actually run at once
    core_count = multiprocessing.cpu_count()
    print(f"Detected {core_count} CPU cores. Initializing 4 stages...")

    start_time = time.perf_counter()

    # Creating 4 separate "Stages" (Processes)
    for i in range(4):
        p = multiprocessing.Process(target=heavy_analysis, args=(i,))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

    end_time = time.perf_counter()
    print(f"Total time with Multiprocessing: {end_time - start_time:.2f} seconds")

Timothy hit enter. For the first time, he heard his laptop's fan kick into high gear immediately. He opened his Activity Monitor and saw four separate rows of Python, each pinned at 100% CPU usage.

"They're actually doing it!" Timothy shouted. "They're all working at the same time!"

The Cost of Independence

"They are," Margaret nodded. "But look at your memory usage. Each of those processes is a full copy of the Python interpreter. They don't share a 'Safe Room' like threads do. If you change a variable in Process 1, Process 2 will never see it. They are living in parallel universes."

Timothy realized the trade-off. By giving everyone their own "Stage" and their own "Microphone," he had achieved True Parallelism. But he had lost the ease of simply sharing a variable.

"So," Timothy mused, "if I want them to talk to each other now, I can't just use a global variable?"

"Exactly," Margaret said. "You'll need a new kind of 'Conveyor Belt' that can cross between universes. But for now, just enjoy the speed. You’ve officially broken the ceiling."

Margaret’s Cheat Sheet: The Multiprocessing Power

When deciding how to handle concurrent tasks, think of the choice between Threads and Processes as a trade-off between shared space and total independence.

Threads: The Lightweight Roommates

Memory: Shared. Imagine everyone working in one room; they can all see the same whiteboard.
The GIL (Global Interpreter Lock): Yes. There is only one microphone in the room. Even if multiple people are there, only one can speak at a time.
Best For: I/O-bound tasks, waiting for web requests, or handling user input.
Resource Usage: Lightweight and fast to start.

Processes: The Independent Specialists

Memory: Isolated. Every worker has their own separate office. They don’t see what the others are doing unless they send a formal email (Inter-process communication).
The GIL: No. Everyone has their own microphone. They can all speak simultaneously without waiting.
Best For: CPU-bound tasks, heavy mathematical calculations, and image/video processing.
Resource Usage: Heavyweight and memory-intensive to manage.

The Specialist's Rules

if __name__ == "__main__": This is mandatory. Without it, your script may try to spawn processes infinitely on Windows/macOS, causing your computer to crash.
cpu_count(): Use multiprocessing.cpu_count() to see how many "Stages" your hardware can actually support.
Isolation: Remember that global variables are not shared. Each process starts with a fresh copy of your script's data.

Aaron Rose is a software engineer and technology writer at tech-reader.blog.

Catch up on the latest explainer videos, podcasts, and industry discussions below.

Search This Blog

Tech-Reader.blog