Sending Multipart Form Data with Python's urllib

When building Python applications that integrate with web services, you may need to send multipart form data, which contains both file uploads and regular form data. Python's standard urllib library provides the tools to handle this.

What is Multipart Form Data?

Multipart form data is used in web forms that have file uploads. Along with the file data, the form also sends regular form data like text fields. The form data is sent in multiple "parts" - one part for the file data, another part for each text field.

The web server reconstructs the parts on the receiving end. This allows files and data to be transmitted together in one request.

Creating a Multipart Form Encoder

Python's urllib library provides the MultipartEncoder to build this multipart data:

import urllib

m = urllib.MultipartEncoder(
    fields={
         # regular form fields
         "text_field": "text value",
         "other_field": "other value",
         # the upload file 
         "file_upload": ("filename", open("file.txt", "rb"), "text/plain")
    }
)

The fields parameter is a dictionary mapping field names to values. For file uploads, use a 3-item tuple with the filename, file handle, and file content type.

Setting the Request Content-Type

When sending multipart form data, the request must include the Content-Type header with the boundary value from the encoder:

import requests

url = "https://api.example.com/upload"

r = requests.post(url, 
    data=m,
    headers={"Content-Type": m.content_type})

The server uses this boundary to distinguish between each part of the multipart data.

Posting Multipart Forms with Requests

The requests library can simplify sending multipart form data. You can directly pass the MultipartEncoder as the data parameter. requests will automatically add the proper Content-Type header:

import requests

url = "https://api.example.com/upload"

r = requests.post(url, data=m) 

Handling Large Uploads

For large file uploads, multipart encoding can consume a lot of memory. An alternative is to send the file data in chunks:

with open("big_file.dat") as f:
    while True:
       chunk = f.read(1024)
       if not chunk: 
           break
       requests.post(url, data=chunk)

This streams the upload directly from disk rather than loading the entire file into memory.

Key Takeaways

The urllib and requests libraries provide the framework for sending robust multipart form data requests from Python. With some care to handle large files, you can build applications that reliably upload files and data to web services.