Multithreading in Python: The threading Module Explained

Starting a thread with the threading module

A thread runs a function alongside the rest of your program instead of after it. You create one by passing a target callable and its arguments to threading.Thread, then call start() to actually launch it:

import threading
import time

def download(name, delay):
    print(f"{name}: starting")
    time.sleep(delay)  # stands in for a slow network call
    print(f"{name}: done")

t1 = threading.Thread(target=download, args=("file-a", 2))
t2 = threading.Thread(target=download, args=("file-b", 2))

t1.start()
t2.start()
t1.join()
t2.join()
print("both downloads finished")

Both threads sleep for two seconds, but because they run at the same time the whole script finishes in roughly two seconds rather than four. join() tells the main thread to wait for a worker to finish before moving on — skip it and your script might print "both downloads finished" while the threads are still running in the background.

Why the GIL limits CPU-bound threading

CPython, the reference implementation almost everyone runs, has a Global Interpreter Lock that only ever lets one thread execute Python bytecode at any instant. Threads still take turns, switching roughly every 5 milliseconds by default (tunable with sys.setswitchinterval()), but only one of them is ever actually running Python at a time. That means a CPU-bound loop split across four threads on a four-core machine does not run four times faster — it usually runs at close to the same speed as a single thread, sometimes slightly slower once you count the overhead of switching:

import threading, time

def count(n):
    while n > 0:
        n -= 1

start = time.perf_counter()
threads = [threading.Thread(target=count, args=(50_000_000,)) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(time.perf_counter() - start)  # not meaningfully faster than one thread

If the bottleneck is genuinely CPU work — number crunching, image processing, parsing large files — multiprocessing sidesteps the GIL by running separate processes instead of threads, each with its own interpreter.

Where threading actually pays off

The GIL is released whenever a thread is waiting on something outside the interpreter: a network response, a disk read, a database query, or a call into a C extension that drops the lock deliberately. That waiting time is exactly where threading earns its keep. Fetching ten URLs one after another might take ten seconds if each request takes about a second; fetching them with ten threads can take close to one second, because the threads spend nearly all their time blocked on the network rather than running Python code:

import threading
import urllib.request

urls = ["https://example.com"] * 10
results = []

def fetch(url):
    with urllib.request.urlopen(url) as resp:
        results.append(resp.status)

threads = [threading.Thread(target=fetch, args=(u,)) for u in urls]
for t in threads: t.start()
for t in threads: t.join()
print(results)

This is the practical rule of thumb: reach for threading when a program spends most of its time waiting, not computing.

Locks and avoiding race conditions

When two threads read and write the same variable without coordination, updates can be lost. Incrementing a counter looks like a single step in Python source, but it compiles to several bytecode operations, and a thread switch can land between them:

import threading

counter = 0

def increment():
    global counter
    for _ in range(200_000):
        counter += 1  # not atomic

threads = [threading.Thread(target=increment) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(counter)  # often less than 800_000

A threading.Lock fixes this by letting only one thread touch the shared value at a time; other threads block on acquire() until it is released:

lock = threading.Lock()

def increment():
    global counter
    for _ in range(200_000):
        with lock:
            counter += 1

The with lock: block acquires the lock on entry and releases it on exit, even if an exception is raised inside — the same context-manager protocol used throughout the standard library. The full set of synchronisation primitives, including RLock and Condition, is documented in the official threading module reference.

ThreadPoolExecutor: a cleaner interface

Creating and joining threads by hand gets repetitive once you have more than a couple of tasks. concurrent.futures.ThreadPoolExecutor manages a fixed pool of worker threads and hands back results as futures:

from concurrent.futures import ThreadPoolExecutor
import urllib.request

def fetch(url):
    with urllib.request.urlopen(url) as resp:
        return resp.status

urls = ["https://example.com"] * 10
with ThreadPoolExecutor(max_workers=10) as pool:
    statuses = list(pool.map(fetch, urls))
print(statuses)

pool.map() preserves input order in its results, dispatches work as workers free up, and the with block waits for every task to finish before continuing — there is no manual bookkeeping of thread objects at all.

Daemon threads and shutdown behaviour

By default, Python waits for every non-daemon thread to finish before the process exits, even after the main thread's code has run to completion. A background thread that polls something forever, like a periodic cache refresher, will keep the whole program alive unless it is marked as a daemon:

t = threading.Thread(target=poll_forever, daemon=True)
t.start()

Daemon threads are killed abruptly when the interpreter exits, with no chance to run cleanup code, so they suit background work you are happy to lose mid-task — not anything writing to a file or holding a lock that needs releasing cleanly. For threads that must shut down in an orderly way, a shared threading.Event checked inside the loop is the safer pattern: set the event, and the thread notices on its next check and exits on its own terms.