S A
Home » Python » Best open-source libraries to make HTTP requests in Python

Best open-source libraries to make HTTP requests in Python

Updated: July 20, 2023 By: Frienzied Flame Post a comment

This concise, straight-to-the-point article will walk you through a list of the best open-source libraries for making HTTP requests (GET, POST, PUT, DELETE, etc) in modern Python.

Table Of Contents
1 httpx
2 requests
3 grequests
4 aiohttp
5 urllib3

httpx

Overview:

This is a modern and async-friendly HTTP client library that supports both traditional sync and modern async requests. It supports features like HTTP/1.1 and HTTP/2, connection pooling, retries, redirects, compression, encoding, SSL verification, authentication, cookies, proxies, timeouts, streaming, and more. It is a relatively new library that was released in 2019, but it has gained popularity and recognition quickly.

Install:

pip install httpx

Code example:

import httpx
import asyncio


async def main():
    async with httpx.AsyncClient() as client:
        response = await client.get(
            "https://api.slingacademy.com/v1/sample-data/photos/1"
        )
        print(f"Response status code: {response.status_code}")
        print(response.text)


asyncio.run(main())

Output:

Response status code: 200
{"success":true,"message":"Photo fetched successfully","photo":{"description":"Leader structure safe or black late wife newspaper her pick central forget single likely.","url":"https://api.slingacademy.com/public/sample-photos/1.jpeg","title":"Defense the travel audience hand","id":1,"user":28}}

Pros:

  • Has a simple and elegant API that is similar to requests but also supports async/await syntax.
  • Has a high performance and scalability that can handle both synchronous and asynchronous requests efficiently.
  • Supports both HTTP/1.1 and HTTP/2 protocols natively and transparently.

Cons:

  • May not be as stable or mature as other libraries that have been around longer.
  • May not be compatible with some third-party libraries or frameworks that expect requests-like objects or interfaces.

requests

Overview:

This is an old, simple, and elegant HTTP library that allows you to send HTTP/1.1 requests extremely easily. It supports features like keep-alive, sessions, cookies, SSL verification, authentication, decompression, file uploads, proxies, timeouts, streaming, etc. It is one of the most downloaded Python packages today, with over 30M downloads per week.

Install:

pip install requests

Code example:

import requests

response = requests.get("https://api.slingacademy.com")
print(f"Status code: {response.status_code}")
print(response.text)

Output:

Status code: 200
{"success":true,"message":"Welcome to Sling Academy Public API"}

Pros:

  • Has a simple and intuitive API that follows the Python philosophy of “batteries included”.
  • Has comprehensive documentation and a large community of users and contributors.
  • Compatible with many third-party libraries and frameworks, such as Django, Flask, urllib3, grequests, etc.
  • Handles most of the common errors and exceptions gracefully and provides useful feedback.
  • Actively maintained and updated by the Python Software Foundation.

Cons:

  • The library doesn’t support async/await. Therefore, it isn’t fast and efficient for some use cases, such as concurrent or asynchronous requests or low-level network operations.
  • May not support some of the newer or less common HTTP features or standards, such as HTTP/2 or WebSockets.

grequests

Overview:

This is a lightweight wrapper around requests and gevent that allows you to send asynchronous requests using greenlets. It supports most of the features that requests supports, such as keep-alive, sessions, cookies, SSL verification, authentication, decompression, file uploads, proxies, timeouts, streaming, etc.

Install:

pip install grequests

Code example:

import grequests

response = (
    grequests.get("https://api.slingacademy.com/v1/sample-data/photos").send().response
)
print(response.status_code)
print(response.text)

Output (just a small part):

200
{"success":true,"message":"Photos fetched successfully","offset":0,"limit":10,"photos":[{"description":"Leader structure safe or black late wife newspaper her pick central forget single likely.","url":"https://api.slingacademy.com/public/sample-photos/1.jpeg"

....

Prop:

  • Provides a simple and elegant API that is almost identical to requests but also supports async/await syntax.
  • Has a low overhead and memory footprint that can handle hundreds of concurrent connections and small amounts of data.

Cons:

  • Doesn’t have good documentation.
  • It may not support some of the newer or less common HTTP features or standards, such as HTTP/2 or WebSockets.
  • It may have some bugs or issues that are not yet resolved or fixed.

aiohttp

Overview:

This is an asynchronous HTTP client and server library that uses the asyncio module to handle concurrency. It supports features like connection pooling, compression, encoding, SSL verification, authentication, cookies, proxies, timeouts, streaming, and more. It is a relatively mature library that was released in 2014, but it has been updated regularly.

Pros:

  • Supports both HTTP/1.1 and HTTP/2 protocols natively and transparently.
  • Has a low-level and flexible API that gives you more control and customization over your HTTP requests.
  • The library supports modern async/await syntax. It has a high performance and scalability that can handle thousands of concurrent connections and large amounts of data.
  • Comprehensive documentation.

Cons:

  • Complex and verbose API that may require more code and effort to use than other libraries.
  • May not be compatible with some third-party libraries or frameworks that expect requests-like objects or interfaces.

You can install aiohttp like so:

pip install aiohttp

Then use it in your code as shown below:

import aiohttp
import asyncio
import certifi

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get(
            "https://api.slingacademy.com"
        ) as response:
            print(response.status)
            print(await response.text())


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Output:

200
{"success":true,"message":"Welcome to Sling Academy Public API"}

urllib3

Overview:

This is a powerful HTTP client library that provides many features that are missing from the standard library urllib module. It brings to the table features like connection pooling, retries, redirects, compression, encoding, SSL verification, authentication, proxies, timeouts, and many more (but rarely used). It is also one of the most downloaded Python packages today, with millions of downloads per week.

Install:

pip install urllib3

Code example:

import urllib3

http = urllib3.PoolManager()
response = http.request("GET", "https://api.slingacademy.com")
print(f"Status: {response.status}")
print(response.data.decode("utf-8"))

Output:

Status: 200
{"success":true,"message":"Welcome to Sling Academy Public API"}

Pros:

  • Gives you a low-level and flexible API that gives you more control and customization over your HTTP requests.
  • Provides a robust error handling and logging system that can help you debug and troubleshoot your HTTP requests.
  • Is widely used and trusted by many other popular libraries and frameworks.
  • Nice documentation.

Cons:

  • Doesn’t support async/await. This is a big drawback in modern Python programming.
  • May not support some of the newer or less common HTTP features or standards

Search tutorials, examples, and resources