Demystifying Authentication with Python Requests

Authentication. It's one of those things that often trips up beginners when working with APIs and web scraping. You try to access a protected resource and get slammed with a 401 Unauthorized error. So you go digging through documentation trying to figure out the proper way to authenticate.

There are so many different types of authentication - basic, digest, OAuth, tokens, etc. It can get confusing pretty quickly. In this article, I'll break down the common authentication schemes supported by Python Requests and show you how to use them properly.

Basic HTTP Authentication

One of the most straightforward authentication schemes is basic HTTP authentication. Here the server just expects a username and password to be included with the request.

Python Requests provides the HTTPBasicAuth class to handle this simply:

from requests.auth import HTTPBasicAuth

auth = HTTPBasicAuth('username', 'password')

r = requests.get(url, auth=auth)

We create a HTTPBasicAuth object with the username and password, and pass that to the auth parameter of the request.

Requests actually makes this even easier - you can just pass a tuple of (username, password) directly to auth without creating the object:

requests.get(url, auth=('username', 'password'))

When would you use basic authentication? Any time an API or website just needs a username/password without greater complexity. It's great for private or internal APIs.

Token-Based Authentication

Many modern APIs use token-based authentication. Rather than passing credentials directly, you get an access token from the API, usually via some authorization workflow. You then pass that token to authenticate.

With Python Requests, you just need to add the token to the Authorization header:

headers = {'Authorization': 'Bearer YOUR_TOKEN'}
requests.get(url, headers=headers)

Replace YOUR_TOKEN with the actual token value provided by the API. The authorization server will validate the token and authenticate the request.

Token auth is nice because the token can expire after a set time, adding improved security. The client just needs to fetch a new token when it expires.

Digest Authentication

Digest auth is a more secure alternative to basic authentication that prevents passwords from being sent in cleartext. It uses a cryptographic hash of the password rather than passing it directly.

Python Requests provides the HTTPDigestAuth class:

from requests.auth import HTTPDigestAuth

auth = HTTPDigestAuth('username', 'password')
requests.get(url, auth=auth)

Digest auth isn't as widely used today due to OAuth, but you may encounter it with certain APIs and web services, especially older ones.

OAuth 1.0a Authentication

OAuth 1.0a is an older authentication standard that still sees some use, particularly with social media APIs like Twitter. It involves a multi-step "handshake" to obtain access tokens.

Requests doesn't handle OAuth 1.0a directly, but the requests-oauthlib library provides the OAuth1 class to make it easy:

from requests_oauthlib import OAuth1

oauth = OAuth1(client_key, client_secret, resource_owner_key, resource_owner_secret)

requests.get(url, auth=oauth)

You pass the various client and access keys to the OAuth1 object and use that for authentication. There is some additional workflow needed to obtain the request and access tokens depending on the OAuth implementation.

OAuth 2.0 and OpenID Connect

OAuth 2.0 is the current industry-standard for authentication and authorization. It is very commonly used by major platforms like Facebook, Google, GitHub, etc. OpenID Connect is an authentication layer built on top of OAuth 2.0.

Once again, the requests-oauthlib library comes to the rescue here with the OAuth2Session class:

from requests_oauthlib import OAuth2Session

client_id = YOUR_CLIENT_ID
client_secret = YOUR_CLIENT_SECRET

oauth = OAuth2Session(client_id, redirect_uri=REDIRECT_URI)
token = oauth.fetch_token(token_url, client_secret=client_secret)

r = oauth.get(url)

This handles the token acquisition workflow and token refresh automatically. You can then call API endpoints normally.

There are various OAuth 2.0 authorization flows - authorization code grant, client credentials, password grant etc. OAuth2Session handles all of these with a consistent interface.

Custom Authentication Logic

For truly custom authentication requirements, Python Requests lets you define your own auth classes by subclassing requests.auth.AuthBase:

from requests.auth import AuthBase

class TokenAuth(AuthBase):
  def __init__(self, token):
    self.token = token

  def __call__(self, r):
    r.headers['X-Token'] = self.token
    return r

You can implement any arbitrary logic inside __call__ to modify the request prior to sending. Return the modified request object at the end.

This is great for proprietary or entirely custom authentication schemes. You can even track state across requests if needed.

Putting it All Together

Between basic, token, digest, OAuth, and custom auth, Python Requests provides a complete authentication solution for calling APIs and web services.

Follow these best practices when working with authentication:

With a good understanding of the available auth classes and how to use them properly, you'll be able to integrate authentication seamlessly into your Python scripts and apps. No more slamming into 401 walls!

So hopefully this article has helped demystify authentication with Python Requests.