Handling 404 Errors when Making HTTP Requests in Python

When making HTTP requests in Python, you may occasionally run into 404 errors, indicating that the URL you tried to request does not exist on the server. Here are some tips on handling these "page not found" errors gracefully in your Python code.

Check for 404 Errors

After making a request using the requests module, check the response status code before trying to use the response content:

import requests

response = requests.get("http://example.com/path")
if response.status_code == 404:
    # Handle 404 error
    print("URL does not exist") 
else:
    # Use response content
    print(response.text)

This checks if the status code in the response is 404 before processing the content.

Log and Notify

Instead of just printing, you may want to log these 404 errors to help debug issues later:

import logging

logger = logging.getLogger(__name__) 

logger.warning("Got 404 for URL %s", url)

You can also notify someone through email or Slack that a URL is returning 404.

Use a Try-Except Block

Wrap your request in a try-except block to handle exceptions gracefully:

try:
    response = requests.get(url)
    response.raise_for_status()
except requests.exceptions.HTTPError as err:
    if err.response.status_code == 404:
        print("404 Error") 

This avoids your program crashing if a URL returns 404.

Handling 404s properly ensures your program does not break unexpectedly. Log and monitor these errors to fix invalid URLs in your code.