Does asyncio use multiple cores python ?

Mar 17, 2024 ยท 2 min read

Python's asyncio module opens up powerful opportunities for parallelism in Python code. But does asyncio actually utilize multiple CPU cores automatically? Let's dig into the details.

Asyncio Provides Concurrency, Not Parallelism

The key thing to understand about asyncio is that it enables concurrency within a single thread, but not parallelism across multiple threads or processes. Here's a quick definitions:

  • Concurrency: Performing multiple tasks over the same period time, but not necessarily simultaneously. Concurrency gives the illusion of parallel execution even when tasks take turns on a single CPU core.
  • Parallelism: Performing multiple calculations literally at the same time, utilizing multiple CPU cores. This is true simultaneous execution.
  • So asyncio enables efficient concurrent code within a single thread, but it does not automatically parallelize code across multiple cores.

    Unlocking Parallelism

    While asyncio itself runs in a single thread, we can utilize multiprocessing or multithreading to achieve parallelism across cores:

    import asyncio
    import multiprocessing
    
    # Function to run in process pool
    def cpu_bound_task(n):
        return sum(i * i for i in range(n))
    
    async def main():
        loop = asyncio.get_running_loop()
        
        # Create a process pool and run tasks
        with multiprocessing.Pool() as pool:
            results = await loop.run_in_executor(
                pool, cpu_bound_task, 1000000)
        
        print(f"Result: {results}")
    
    asyncio.run(main())

    Here we use run_in_executor to dispatch CPU-bound work to a process pool, while keeping the main asyncio thread free. The key takeaway is that with a little extra work we can utilize multiple processes or threads to achieve true parallelism.

    When to Use Asyncio

    Asyncio shines for I/O-bound work like network calls, file operations, and interfacing with async databases. For pure CPU-bound number crunching, multiprocessing may be better suited. Evaluate your specific use case.

    The advantage of asyncio is it enables concurrency using simple sequential code, avoiding callback- or promise-based code. For I/O and mixed workloads, that makes it very appealing!

    I hope this gives some clarity on how asyncio enables concurrency but not inherent parallelism. With the right approach, we can utilize asyncio alongside multiprocessing or multithreading for parallel execution when needed!

    Browse by tags:

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you


    Try ProxiesAPI for free

    curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

    <!doctype html>
    <html>
    <head>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />
    ...

    X

    Don't leave just yet!

    Enter your email below to claim your free API key: