The Secret Life of Python: The Conveyor Belt

Thread-safe communication with queues

#Python #Threading #Concurrency #SoftwareEngineering

Margaret is a senior software engineer. Timothy is her junior colleague. They work in a grand Victorian library in London — the kind of place where code quality is the unspoken objective, and craftsmanship is the only thing that matters.

Episode 34

Timothy was exhausted. He had spent the week managing locks, checking acquisition orders, and setting timeouts to prevent deadlocks.

"Margaret," Timothy said, "locking everything is like trying to run a restaurant where only one person is allowed in the kitchen at a time, and they have to lock the fridge, the stove, and the sink separately. It’s making my head spin. Is there a way to just... hand off the work?"

Margaret smiled and walked over to the whiteboard. She didn't draw a safe or a vault this time. She drew a long, moving belt.

"You’ve been fighting over Shared Data, Timothy. That’s hard. Instead, you should try Message Passing. In Python, we do that with a Queue."

The Producer and The Consumer

"Imagine a factory," Margaret said. "On one side, you have the Producer (your match engine). It finishes a move and puts it in a box. It places that box on a Conveyor Belt (The Queue) and goes back to work. It doesn't care who picks it up."

"On the other side," she continued, "you have the Consumer (your logger). It sits at the end of the belt. The moment a box arrives, it picks it up, writes it to the file, and waits for the next one."

"The best part?" Margaret added. "The belt is Thread-Safe. Python handles all the locking internally. You don't need a single with lock: statement to use it."

Putting it on the Belt

Timothy looked at the code. It was shockingly clean.

import threading
import queue
import time

# Our Conveyor Belt (limited to 10 items to prevent overflow)
match_queue = queue.Queue(maxsize=10)

def producer():
    for i in range(5):
        move = f"Move {i}: e4"
        print(f"Producer: Putting '{move}' on the belt.")
        match_queue.put(move) # Handing off the work
        time.sleep(0.5)

def consumer():
    while True:
        # The Consumer waits until something is on the belt
        try:
            move = match_queue.get(timeout=2) 
            print(f"Consumer: Processing '{move}' and saving to file.")
            match_queue.task_done() # Signaling the work is finished
        except queue.Empty:
            print("Consumer: Belt is empty, taking a break...")
            break

# Start the workers
threading.Thread(target=producer, daemon=True).start()
threading.Thread(target=consumer, daemon=True).start()

# Let the match run for a bit
time.sleep(4)
print("Match Finished.")

"Wait," Timothy said. "I didn't have to lock the match_queue. I just put() and get()?"

"Exactly," Margaret said. "The Queue is a Specialist that handles its own locks. By using it, you’ve decoupled your threads. The Producer doesn't have to wait for the Logger to finish writing before it calculates the next move. They just communicate through the belt."

The Specialist’s Peace

Timothy watched the moves flow smoothly from one thread to the other. There were no race conditions. There were no deadlocks. By changing the architecture from "Fighting over a Variable" to "Passing a Message," the chaos had turned into a rhythm.

"It’s not just about safety," Timothy realized. "It’s about flow."

Margaret’s Cheat Sheet: The Power of Queues

The Core Pattern

The Queue: A First-In, First-Out (FIFO) data structure that is natively thread-safe.
Producer-Consumer: A design where one thread creates work and another thread processes it.
put() and get(): These methods handle all the "under the hood" locking for you. By default, they will "block" (wait) until space or an item is available.

The Specialist's Wisdom

Decoupling: Threads no longer need to know about each other’s internal state. They only need to know about the belt.
Flow Control: Use maxsize to ensure your Producer doesn't overwhelm your Consumer and fill up your computer's memory.
Lock vs. Queue: Use Locks to protect a few simple variables. Use Queues to pass work or messages between different parts of your program.

Aaron Rose is a software engineer and technology writer at tech-reader.blog.

Catch up on the latest explainer videos, podcasts, and industry discussions below.

Search This Blog

Tech-Reader.blog