Streaming Uploads in Python Requests using File-Like Objects

When making requests in Python, you may sometimes need to upload large files like videos, audio, or other binary data. Rather than loading the entire file contents into memory, you can upload data in chunks using a file-like object.

This saves memory and allows streaming uploads even for very large files.

Why Use File-Like Objects?

Uploading data using the files argument in Requests loads the full contents of the file into memory. This can cause problems for large files:

with open('large_video.mp4', 'rb') as f:
    requests.post('https://example.com/upload', files={'video': f}) 

Instead, we can create a file-like object that streams the data in chunks:

with open('large_video.mp4', 'rb') as f:
    requests.post('https://example.com/upload', data=f)

This streams the file data in memory-efficient chunks without loading the entire file.

Creating a File-Like Object in Python

Many built-in Python file objects already support file-like streaming uploads:

with open('data.bin', 'rb') as f:
    requests.post(url, data=f)

You can also create a custom file-like object by implementing Python's file-like API:

import io

class StreamingBody:
    def __init__(self, filepath):
        self.filepath = filepath
    
    def read(self, n):
        # custom read implementation
        pass 

    def seek(self, offset):
        # custom seek implementation
        pass

stream = StreamingBody('video.mp4')
requests.post(url, data=stream)

When to Avoid Streaming Uploads

Streaming file uploads add complexity and may not always be necessary. For smaller files under 10-20MB, avoid complex file streams.

Just loading the contents directly with the files argument works great:

with open('report.pdf', 'rb') as f:
   requests.post(url, files={'report': f})

So in summary, file-like streaming objects allow efficient upload of large binary data in Python Requests while avoiding high memory usage. Implement the file-like API on any object to enable streaming uploads.