Automate Search Form Submission with Python Requests

Submitting forms is a common task when scraping the web or automating workflows. Manually filling out forms over and over wastes time. Python requests allows you to easily submit forms programmatically.

Submitting Search Forms

Many websites have search forms to filter content. For example, an ecommerce site may have a search box to find products. Python requests can automatically submit these forms so you can retrieve the filtered results in your code.

Here is an example search form:

<form action="/search" method="get">
  <input type="text" name="query">
  <input type="submit" value="Search">
</form>

To submit this form in Python:

import requests

url = 'https://www.example.com/search'
payload = {'query': 'hello world'}

r = requests.get(url, params=payload)
print(r.url)
# https://www.example.com/search?query=hello+world

The key things to note:

This automatically encodes and submits the form data. The response contains the results page based on the submitted search terms!

Tips

Learning to programmatically submit forms opens up many possibilities for web scraping and automation with Python requests!