Debugging HTTP Requests in Python with Request Logging

When working with the Python requests library to make HTTP requests, it can be invaluable to log the full request and response to debug issues. Here are some simple techniques to add comprehensive logging to your Python requests scripts.

Log Entire Request and Response to Console

The quickest way to log requests and responses is to use the built-in logging in requests. Add this before making the request:

import logging
import requests 
requests.packages.urllib3.add_stderr_logger()

This will print the full URL, headers, and body of both requests and responses to the console.

Log to File for Production

For apps in production, log to a file instead of console:

import logging
logging.basicConfig(filename='app.log', level=logging.DEBUG)
requests.packages.urllib3.add_stderr_logger() 

This will append the logs to app.log for later debugging.

Log Specific Parts of Requests/Responses

To log specific parts like URL, status code, headers, or body, use the Python logging library:

import requests
import logging

logger = logging.getLogger('requests_logger')

r = requests.get('https://api.example.com/items')
logger.debug(f'URL: {r.url}')
logger.debug(f'Status Code: {r.status_code}')
logger.debug(f'Headers: {r.headers}')
logger.debug(f'Body: {r.json()}')

This allows logging exactly what you need from requests without getting overwhelmed by too much data.

Conclusion

Adding logging to Python requests provides visibility into issues when making HTTP requests. Use the built-in stderr logging for quick debugging, log to a file for production, and log specific attributes like URL, status and headers for targeted debugging. Proper request/response logging can save hours of time troubleshooting Python apps.