Post

Threading in python

A beginner-friendly guide to threading in Python. Understand how to run multiple threads, manage concurrency, and improve program performance with practical examples.

Threading in python

Python’s threading module is a powerful tool for running tasks concurrently, especially when dealing with I/O-bound operations. However, it comes with certain nuances—particularly the Global Interpreter Lock (GIL)—that every developer should understand.

This guide consolidates essential notes, examples, and tricks for working with threads in Python.


1. Understanding Threads in Python

Threads in Python are lightweight processes managed by the operating system. When a program spawns a thread, it hands control over to the OS to execute it.

Key Points:

  • Global and Local Variables in Threads
    • Global variables can be accessed from any thread.
    • To modify a global variable inside a thread, declare it with the global keyword; otherwise, Python treats it as a local variable.
  • Target Functions
    • A callable function can be passed to a thread’s target parameter.
    • The run() method is the default execution method of a thread. Overriding it allows custom thread behavior.

2. The GIL (Global Interpreter Lock)

In CPython:

  • The GIL ensures that only one thread executes Python bytecode at a time.
  • Implication: True parallelism is not achieved for CPU-bound tasks, but I/O-bound operations still benefit from threading.
  • Benefit: Reduced risk of data corruption without locks (since threads don’t truly run at the exact same time in CPU-bound tasks).
  • Drawback: No real CPU-level parallelism for computational tasks.

In Other Implementations:

  • Jython (Java-based Python) and some others do not have a GIL, achieving true parallelism.
  • PyPy is a faster implementation but still has a GIL.

3. When to Use Threading

  • Ideal for I/O-bound tasks (e.g., network calls, file I/O).
  • Not suitable for CPU-heavy computation—use multiprocessing for those.

4. How threading Works

The threading.Thread class handles thread creation and management.

  • Constructor (__init__):
    • Accepts a callable via target.
    • args can be used to pass parameters to the callable.
  • start() method:
    • Invokes the run() method in a new thread.
  • Overriding run():
    • Lets you define custom behavior for the thread.
  • Access to Global Namespace:
    • Threads share the main namespace, so functions/variables from the main thread are accessible.

5. Thread Creation Methods

Method 1: Using target with a function

1
2
3
4
5
6
7
import threading

def thread_fun(value):
    print("Thread called with value:", value)

t1 = threading.Thread(target=thread_fun, args=(123,))
t1.start()

Method 2: Inheriting Thread and Overriding run()

1
2
3
4
5
6
7
8
9
import threading

class MyThread(threading.Thread):
    def run(self):
        result = 1 + 1
        print("Running custom logic. Result:", result)

t1 = MyThread()
t1.start()

Method 3: Overriding both __init__ and run()

1
2
3
4
5
6
7
8
9
10
11
12
import threading

class MyThread(threading.Thread):
    def __init__(self, value):
        super().__init__()
        self.value = value
    
    def run(self):
        print("Value passed to thread:", self.value)

t1 = MyThread(1234)
t1.start()

6. Useful Threading Patterns & Tricks

Thread Pooling with a List

1
2
3
4
5
6
7
8
9
10
11
12
13
import threading

def worker():
    print("Working...")

threads = []
for _ in range(5):
    t = threading.Thread(target=worker)
    t.start()
    threads.append(t)

for t in threads:
    t.join()

Thread Synchronization with Locks

When multiple threads modify a shared resource, use locks to prevent race conditions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import threading

lock = threading.Lock()
shared_value = 0

def increment():
    global shared_value
    with lock:
        shared_value += 1

threads = [threading.Thread(target=increment) for _ in range(10)]
for t in threads: t.start()
for t in threads: t.join()

print("Final value:", shared_value)

Using Events for Coordination

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import threading
import time

event = threading.Event()

def waiter():
    print("Waiting for event...")
    event.wait()
    print("Event received!")

t = threading.Thread(target=waiter)
t.start()

time.sleep(2)
event.set()

Alarm Thread Example

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import os, time, threading

class Alarm(threading.Thread):
    def __init__(self, timeout):
        super().__init__()
        self.timeout = timeout

    def run(self):
        time.sleep(self.timeout)
        os._exit(0)  # Force exit

alarm = Alarm(4)
alarm.start()

time.sleep(3)
print("Main process still running...")

threading.local() ?

It creates an object where each thread can store and access its own attributes without interfering with other threads.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import threading
import time

# Create thread-local storage
local_context = threading.local()

def worker(id):
    setattr(local_context ,threading.current_thread().name,f"Thread-{id}-data")
    # local_context.value = f"Thread-{id}-data"      or use this way
    time.sleep(2)
    print(local_context.__dict__)
    # print(f"[{threading.current_thread().name}] value = {thread_local_data.value}")

threads = []

for i in range(3):
    t = threading.Thread(target=worker, args=(i,))
    threads.append(t)
    t.start()
setattr(local_context ,threading.current_thread().name,f"This is main thread")

print(local_context.__dict__)

for t in threads:
    t.join()

Each thread sees its own value, even though it’s using the same thread_local_data object. Under the hood, threading.local() manages this separation.

Why Use threading.local()?

  • Avoids global state conflicts
  • Each thread gets its own isolated data

Very useful in:

  • ✅ Logging contexts
  • ✅ Request/response data in threaded servers
  • ✅ Database session management per thread

✅ When NOT to Use It

  • ❌ In async code (like asyncio) → use contextvars instead
  • ❌ If your workload is not thread-based (e.g., uses multiprocessing)

⚠️ Notes

  • threading.local() is not a dictionary — you assign attributes to it:

    1
    2
    3
    4
    5
    6
    
      import threading
    
      local_data = threading.local()
      local_data.user_id = 123
      setattr(local_data,"user_id",123)
      # Both ways are possible
    
  • Don’t use threading.local() in async code — it will leak data between coroutines.
  • contextvars is available from Python 3.7+.
  • If you’re doing both threads and async (e.g., using asyncio inside threads), you’ll need to understand both tools and how they interact.
  • You cannot share data between threads with it — that defeats the purpose.

7. Threading vs Multiprocessing

FeatureThreadingMultiprocessing
ParallelismLimited by GIL in CPythonTrue parallelism
Best forI/O-bound tasksCPU-bound tasks
Memory UsageLightweightHeavier (separate processes)
CommunicationShared memory (care with locks)IPC mechanisms (queues, pipes)

8. Final Tips

  • Use threading for network I/O, file I/O, and waiting tasks.
  • Use multiprocessing for number crunching and CPU-intensive tasks.
  • Always remember: start() runs the thread, run() contains the logic, and join() waits for completion.
  • Avoid unnecessary complexity—keep threaded code simple to prevent hard-to-debug issues.

References:

This post is licensed under CC BY 4.0 by the author.