How to use requests with asyncio python?

In Python, we use asyncio library for writing the concurrent code by using the aysnc/await syntax. However, while performing the HTTP requests asynchronously, the requests library does not support asyncio by default.

In this article, we will see how to use requests with asyncio for sending HTTP requests asynchronously.

How requests is different from asyncio?

requests is a synchronous HTTP library, which means it blocks the execution of the code until the HTTP request is completed. But the behaviour of asyncio is opposite to it.

Requests in a Thread Pool

requests can be used along with asyncio by running it in a separate thread. Using thread pool’s loop.run_in_executor we can run asynchronous functions in a thread pool which allows requests call to be executed asynchronously.

import asyncio
import requests
from concurrent.futures import ThreadPoolExecutor

async def fetch(url):
    with ThreadPoolExecutor() as executor:
        loop = asyncio.get_event_loop()
        response = await loop.run_in_executor(executor, requests.get, url)
        return response.text

async def main():
    url = "http://example.com"
    html = await fetch(url)
    print(html)

asyncio.run(main())

In the above example, fetch is an asynchronous function that uses requests.get to fetch a webpage’s HTML. It runs in a thread pool so that it doesn’t block other asynchronous operations.

Asynchronous HTTP libraries

As an alternative way, we can also use the libraries which are designed to work with asyncio. Libraries like aiohttp are asynchronous HTTP clients for asyncio.

Following is the example:

import asyncio
import aiohttp

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    async with aiohttp.ClientSession() as session:
        url = "http://example.com"
        html = await fetch(session, url)
        print(html)

asyncio.run(main())

In the above example, aiohttp is used for making the HTTP request asynchronously. It is more efficient and suitable for asyncio-driven applications.

Frequently asked questions

1. Can I use requests directly with asyncio without a thread pool?
   No, requests is a synchronous library and will block the asyncio event loop.
2. Is aiohttp a complete replacement for requests?
   aiohttp is powerful for async operations but may lack some convenience features of requests for synchronous HTTP calls.
3. Are there any performance benefits to using aiohttp over requests in a thread pool?
   Yes, aiohttp is generally more efficient for asynchronous operations as it is non-blocking and designed to be used with asyncio.