How to write URL in Python?

When working with web applications, APIs, or scraping websites in Python, you'll inevitably need to handle URLs. Here are some best practices for writing, encoding, parsing, and modifying URLs in Python.

Writing URLs

You can specify URLs in Python using normal strings like:

url = "https://www.example.com/path?key=value" 

Prepend http:// or https:// for absolute URLs. You can also construct URLs programmatically by concatenating strings:

root_url = "https://www.example.com"
path = "/products"
url = root_url + path

Encoding URLs

When including URLs in queries or path segments, ensure special characters are URL encoded using the urlencode() function from urllib:

from urllib.parse import urlencode
params = {"q": "green eggs & ham"} 
print(urlencode(params)) # q=green+eggs+%26+ham

This converts characters like &, =, and spaces to percent-encoded characters like %26.

Parsing URL Components

To manipulate a URL's protocol, hostname, path etc individually, use urllib's urlparse() method:

from urllib.parse import urlparse

url = "http://user:[email protected]/path?query=true"
parsed = urlparse(url)

print(parsed.scheme) # 'http' 
print(parsed.hostname) # 'example.com'

You can modify components of the ParseResult and rebuild with urlunparse().

Practical Usage

When making HTTP requests with the requests module, it will automatically handle correct URL encoding and parsing for you. But when scraping websites or handling URLs for an API, utilize Python's URL handling utilities like urlencode() and urlparse() to ensure your URLs are valid.

The key is using the right tool for the job - Python makes handling URLs easy, as long as you leverage the built-in libraries.