Introduction

Working with HTTP requests in Python often means using the excellent Requests library. It provides a simple interface for sending requests and handling responses.

However, sometimes you need more visibility into what Requests is doing under the hood. Logging and debugging capabilities become essential for troubleshooting issues or just understanding the full request/response flow.

In this comprehensive guide, we'll explore various techniques for enabling detailed logging and debugging with Requests.

Overview of Logging and Debugging with Requests

The Requests library uses several internal components for handling HTTP connections, including:

To enable detailed logging, we need to configure both of these components.

The key goals are:

Requests doesn't have great built-in logging, so we need to dig into the internals to enable it.

When to Enable Debugging

You wouldn't want verbose debugging enabled in production. But during development and testing, it becomes indispensable.

Some common use cases:

Debug logs allow offline analysis. You can save them to files and dig in later if issues crop up.

Enabling logging in your integration tests provides ongoing visibility into API calls.

Built-in Logging

Let's start with Requests' built-in logging capabilities.

Using urllib3 Logger

The urllib3 library handles HTTP connections in Requests. We can enable debug logs by configuring its logger.

Importing and Configuring Logger

First import the logger and set the log level to DEBUG:

import logging
log = logging.getLogger('urllib3')
log.setLevel(logging.DEBUG)

This will enable detailed debug logging from urllib3.

Setting Log Level

To further refine the logging, you can set the severity threshold.

For example, to show warnings and above:

log.setLevel(logging.WARNING)

The hierarchy from most verbose to least is:

Printing Requests and Responses

With debug logging enabled, you can see the full request and response details:

DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): example.com:80
DEBUG:urllib3.connectionpool:<http://example.com:80> "GET / HTTP/1.1" 200 3256

This prints the method, URL, protocol, status code, and response size.

Setting HTTPConnection Debug Level

For even more verbosity, we can enable debugging in the http.client module that urllib3 uses under the hood:

Importing HTTPConnection

Import HTTPConnection from http.client:

from http.client import HTTPConnection

Setting debuglevel

Set the debuglevel to 1 to enable debugging:

HTTPConnection.debuglevel = 1

Example Debug Output

Now debugs will show the request headers and body:

DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): example.com:80
DEBUG:send: b'GET / HTTP/1.1
Host: example.com
User-Agent: python-requests/2.22.0

'
DEBUG:reply: 'HTTP/1.1 200 OK\\r\\n'
DEBUG:header: Server: nginx
DEBUG:header: Content-Type: text/html
DEBUG:header: Content-Length: 1234
DEBUG:urllib3.connectionpool:<http://example.com:80> "GET / HTTP/1.1" 200 1234

This gives us complete request/response details from the built-in logging.

Basic Logging Configuration

For more control over logging, we can configure it directly instead of using the built-in options.

Importing Logging Module

Import Python's logging module:

import logging

This will let us configure loggers and handlers.

Setting Root Logger Level

Set the global log level on the root logger:

logging.basicConfig(level=logging.DEBUG)

This enables debug logging globally.

Printing Debug Messages

We can now print log messages:

logger = logging.getLogger(__name__)

logger.debug("Request headers: %s", headers)
logger.debug("Request body: %s", body)

The messages will be emitted because of the debug log level.

Custom Logging

For more advanced usage, we can customize formatting, utilize hooks, and route logs.

Formatted Output

Basic logging just prints plaintext messages. For readability, we can format the output.

Creating Custom Formatter

Define a formatter class to add structure:

class RequestFormatter(logging.Formatter):

    def format(self, record):
        message = super().format(record)
        # Add custom formatting
        return message

Configuring Handler and Formatter

Next configure a handler to use the formatter:

handler = logging.StreamHandler()
handler.setFormatter(RequestFormatter())

Printing Formatted Logs

Logging through the handler will now be formatted:

logger = logging.getLogger(__name__)
logger.addHandler(handler)

logger.info("Formatted request log")

Logging Hooks

Hooks allow tapping into Requests before and after requests. We can use them to log.

Request Hook

Define a hook function to log on requests:

def log_request(request):
    logger.debug("Request: %s", request)

Response Hook

Similarly, log after responses:

def log_response(response):
    logger.debug("Response: %s", response)

Request-Response Roundtrip

Putting it together to log the full roundtrip:

session = requests.Session()
session.hooks["request"].append(log_request)
session.hooks["response"].append(log_response)

Printing via Hooks

Now any requests through this session will be logged:

session.get("<http://example.com>")

This generates:

Request: <PreparedRequest [GET]>
Response: <Response [200]>

Hooks give us fine-grained control to log events.

Advanced Configuration

There are a few other useful logging configurations.

Logging to File

For persistent storage, write logs to a file:

file_handler = logging.FileHandler("debug.log")
logger.addHandler(file_handler)

This will save logs instead of printing to stdout.

Logging to Stdout

To print to standard output instead of stderr:

stdout_handler = logging.StreamHandler(sys.stdout)
logger.addHandler(stdout_handler)

Disabling Logging

Disable debugging with:

logger.setLevel(logging.WARNING)

Increase the threshold to suppress less important messages.

Third-Party Libraries

For more advanced logging, consider dedicated libraries like loguru. They provide much easier configuration and formatting.

Conclusion

Summary of Techniques

In this guide, we covered various techniques for debugging Requests:

Recommendations

Careful logging will provide invaluable visibility into your Requests usage. Following these patterns will ensure you have the right level of verbosity when issues arise or you need to audit request handling in your systems.