Home

repost: Introduction to concurrent.futures in Python - smrati katiyar

Concurrency in Python can be efficiently handled using the concurrent.futures module. This module provides a high-level interface for asynchronously executing function calls using either threads or processes. It allows you to run tasks in parallel, whether they are I/O-bound or CPU-bound, without worrying too much about the underlying details of thread and process management.

In this article, we’ll explore how to use concurrent.futures , explain the differences between threads and processes, and provide examples to illustrate the concepts.

# What is concurrent.futures ?

The concurrent.futures module provides two primary classes for parallel execution:

  • ThreadPoolExecutor: Uses threads to perform tasks. Best suited for I/O-bound operations where tasks spend time waiting (e.g., file I/O or network requests).
  • ProcessPoolExecutor: Uses separate processes to perform tasks. Ideal for CPU-bound tasks, such as computation-heavy operations, since Python’s Global Interpreter Lock (GIL) does not apply to separate processes.

Both classes work similarly and allow you to submit tasks for concurrent execution, providing a way to execute them in parallel or asynchronously while managing their results.

# Key Concepts

  • Executor: An executor is an object that manages worker threads or processes.
  • Future: A Future object represents a result that may not have been computed yet. It allows you to check if the task is complete, retrieve the result, or cancel the task if needed.
  • submit: The submit() method is used to schedule a function for execution and returns a Future object.
  • map: Similar to Python’s built-in map() , but tasks are executed concurrently.

Let’s dive into examples to see how each component works.

# 1. Using ThreadPoolExecutor for I/O-Bound Tasks

The ThreadPoolExecutor is perfect for I/O-bound tasks like reading files, making network requests, or handling user input concurrently.

# Example: Downloading multiple web pages concurrently

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import concurrent.futures
import time
import requests

# Function to download a webpage
def download_page(url):
    response = requests.get(url)
    return f"{url} - {len(response.content)} bytes"

# List of URLs to download
urls = [
    "https://www.example.com",
    "https://www.python.org",
    "https://www.github.com",
]

# Using ThreadPoolExecutor to download pages concurrently
start_time = time.time()

with concurrent.futures.ThreadPoolExecutor() as executor:
    futures = [executor.submit(download_page, url) for url in urls]
    for future in concurrent.futures.as_completed(futures):
        print(future.result())

end_time = time.time()
print(f"Downloaded all pages in {end_time - start_time:.2f} seconds")

# Explanation:

  • The download_page() function fetches a webpage and returns the size of the page content.
  • Using ThreadPoolExecutor , we submit the tasks to download each URL concurrently.
  • The concurrent.futures.as_completed() function returns each Future as it completes, allowing us to process results as soon as they are ready.

# Output:

1
2
3
4
https://www.python.org - 48730 bytes
https://www.example.com - 1256 bytes
https://www.github.com - 30773 bytes
Downloaded all pages in 1.23 seconds

In this example, tasks that could take time to wait for network responses are handled in separate threads, speeding up the overall execution time.

# 2. Using ProcessPoolExecutor for CPU-Bound Tasks

For CPU-intensive tasks like mathematical computations, ProcessPoolExecutor is more suitable. Python’s GIL (Global Interpreter Lock) limits true parallelism in threads, but since processes have separate memory spaces, multiple processes can run in parallel on different CPU cores.

# Example: Parallel computation using ProcessPoolExecutor

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import concurrent.futures
import time

# Function to perform a CPU-bound task
def compute_factorial(n):
    result = 1
    for i in range(2, n + 1):
        result *= i
    return result

numbers = [50000, 60000, 70000, 80000]

# Using ProcessPoolExecutor to compute factorials concurrently
start_time = time.time()

with concurrent.futures.ProcessPoolExecutor() as executor:
    futures = [executor.submit(compute_factorial, num) for num in numbers]
    for future in concurrent.futures.as_completed(futures):
        print(f"Factorial computed for number: {numbers[futures.index(future)]}")

end_time = time.time()
print(f"Computed all factorials in {end_time - start_time:.2f} seconds")

# Explanation:

  • The compute_factorial() function performs a CPU-bound operation of calculating the factorial of a number.
  • Using ProcessPoolExecutor , we submit each factorial computation task to be executed in a separate process.
  • The concurrent.futures.as_completed() function is used to retrieve results as they become available.

Output

1
2
3
4
5
Factorial computed for number: 50000
Factorial computed for number: 60000
Factorial computed for number: 70000
Factorial computed for number: 80000
Computed all factorials in 2.98 seconds

This example demonstrates how CPU-bound tasks can be handled in parallel across multiple CPU cores, improving performance.

# 3. Using map for Simple Concurrent Task Execution

Both ThreadPoolExecutor and ProcessPoolExecutor provide a convenient method called map() to apply a function to multiple inputs concurrently. It works similarly to Python’s built-in map() , but with concurrent execution.

# Example: Using map with ThreadPoolExecutor

1
2
3
4
5
6
7
8
9
10
11
12
13
import concurrent.futures

def square(n):
    return n * n

numbers = [1, 2, 3, 4, 5]

# Using ThreadPoolExecutor to calculate squares concurrently
with concurrent.futures.ThreadPoolExecutor() as executor:
    results = executor.map(square, numbers)

# Results are returned in the order of input
print(list(results))

Output

1
[1, 4, 9, 16, 25]

Here, executor.map() applies the square() function to each element in numbers concurrently. The map() function returns results in the same order as the input data, making it a convenient choice for cases where task order matters.

# 4. Handling Exceptions in Futures

When working with concurrent tasks, some tasks may fail or raise exceptions. concurrent.futures allows you to handle these exceptions gracefully.

# Example: Exception Handling with Future

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import concurrent.futures

def faulty_task(n):
    if n == 2:
        raise ValueError("Error with input 2")
    return n * n

numbers = [1, 2, 3, 4]

with concurrent.futures.ThreadPoolExecutor() as executor:
    futures = [executor.submit(faulty_task, num) for num in numbers]
    for future in concurrent.futures.as_completed(futures):
        try:
            result = future.result()  # Retrieve result or raise an exception
            print(f"Result: {result}")
        except Exception as e:
            print(f"Task raised an exception: {e}")

Output

1
2
3
4
Result: 1
Task raised an exception: Error with input 2
Result: 9
Result: 16

In this example, the faulty_task() function raises an exception for input 2 . The future.result() method either returns the result of the task or raises an exception if the task failed.

# Conclusion

The concurrent.futures module in Python offers a powerful and simple interface for writing concurrent and parallel code. Whether you’re dealing with I/O-bound tasks using ThreadPoolExecutor or CPU-bound tasks with ProcessPoolExecutor , this module provides a high-level way to manage tasks and retrieve results asynchronously.

By combining concepts like Future , submit() , and map() , you can easily implement concurrency into your Python programs and achieve significant performance improvements for both I/O-bound and CPU-bound operations.

Edited on