Accessing HTTPS Sites with Self-Signed Certs in Python Requests

When accessing HTTPS URLs in Python using the Requests module, you may encounter SSL certificate verification errors if the site uses a self-signed certificate rather than one signed by a trusted certificate authority. This poses a challenge for testing and development scenarios.

Fortunately, the Requests module provides options to handle these self-signed certificate cases in a secure way. Here we'll explore a few methods to safely access HTTPS sites that use self-signed certificates with Requests.

The Risks of Ignoring SSL Errors

First it's important to understand why manually disabling SSL certificate verification is risky. Doing something like:

import requests
requests.get('https://self-signed.example.com', verify=False)

This exposes your application to man-in-the-middle (MITM) attacks where data can be intercepted or manipulated. So disabling verify is not recommended in production.

Using certifi to Load Root Certificates

The easiest way to handle self-signed certs is including the certifi package along with Requests. This contains root certificates from Mozilla to validate certificate chains.

Just install certifi and Requests will use it automatically:

pip install requests certifi

This allows Requests to validate certificate chains while still allowing you to access sites with self-signed leaf certificates.

Supplying Your Own Certificate Bundle

Alternatively, you can provide your own custom bundle containing the self-signed cert:

import requests

cert_path = '/path/to/self-signed-cert.pem' 

requests.get('https://self-signed.example.com', verify=cert_path)

This is useful for cases where you want to test with fully custom certs not covered by root stores like certifi.

Using REQUESTS_CA_BUNDLE Environment Variable

Setting the REQUESTS_CA_BUNDLE environment variable to your custom cert path also works:

export REQUESTS_CA_BUNDLE=/path/to/self-signed-cert.pem
python script.py

This approach sets the trusted certs globally without coding it for each script.

Wrapping the SSL Context

For more advanced cases, you can wrap the default SSL context to add certs or adjust other SSL options:

import ssl
import requests

ssl_context = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH, cafile='/path/to/self-signed-cert.pem')
requests.get('https://self-signed.example.com', verify=False, ssl_context=ssl_context)

Here we handle the cert verification at the SSL context level.

Development Server Certificates

For development/testing it's also common to use automation tools like Ansible, Puppet or Docker to automatically generate and install certs on local servers to avoid manual steps.

These tools typically allow exporting the CA certificate to distribute to clients. You can feed this CA cert bundle into one of the above Requests options to complete the chain of trust.

Summary

In this article we covered several methods to securely access HTTPS-enabled sites using self-signed certificates with Python Requests:

Taking advantage of these options allows accessing HTTPS test servers during development while still maintaining security.

The key takeaway is to always validate certificate chains rather than disabling verification outright. This prevents man-in-the-middle attacks when working with self-signed certs.