Streaming Downloads with Python Requests

When downloading files in Python using the requests library, you may want to stream the response body instead of loading the entire file contents into memory at once. Streaming the response allows you to handle large downloads without running out of memory, and start processing the data before the download completes.

Why Stream Downloads?

Normally when you call response.content, requests will load the entire response body into memory. This is fine for small downloads, but can cause issues with larger files.

Streaming the response with response.iter_content() iterates over the content in chunks, so you can handle large downloads without buffering everything in memory.

Benefits include:

Streaming Example

Here's how to stream a download while printing progress to the console:

import requests
import shutil

url = 'http://example.com/large_file.zip'

r = requests.get(url, stream=True)

with open('large_file.zip', 'wb') as f:
    for chunk in r.iter_content(chunk_size=1024*8): 
        if chunk:
            print(f"Downloading {r.headers.get('content-length')} bytes...") 
            f.write(chunk)

The key points are:

This streams the download while printing progress. We handle the response in chunks instead of one large buffer for lower memory usage.

Handling Compressed Streams

For compressed streams like gzip/deflate, use iter_lines() instead to decode the compressed stream. Setting decode_unicode=True will properly handle character encodings as well.

This provides a simple way to stream large downloads in Python while avoiding buffering entire file contents in memory!