Downloading Images from a Website with Elixir and Floki

In this article, we will learn how to use Elixir and the HTTPoison and Floki libraries to download all the images from a Wikipedia page.

—-

Overview

The goal is to extract the names, breed groups, local names, and image URLs for all dog breeds listed on this Wikipedia page. We will store the image URLs, download the images and save them to a local folder.

Here are the key steps we will cover:

1. Import required modules
2. Send HTTP request to fetch the Wikipedia page
3. Parse the page HTML using Floki
4. Find the table with dog breed data using a CSS selector
5. Iterate through the table rows
6. Extract data from each column
7. Download images and save locally
8. Print/process extracted data

Let's go through each of these steps in detail.

Imports

We need these modules:

import HTTPoison
import Floki

Send HTTP Request

To download the web page:

url = "<https://commons.wikimedia.org/wiki/List_of_dog_breeds>"

response = HTTPoison.get!(url, [], hackney: [pool: :default, recv_timeout: 25000])

We make a GET request to the URL.

Parse HTML

To parse the HTML:

html = response.body |> Floki.parse_document!()

The html var now contains the parsed document.

Find Breed Table

We use a CSS selector to find the table element:

table = html |> Floki.find("table.wikitable.sortable")

This selects the

Iterate Through Rows

We loop through the rows like this:

Floki.find(table, "tr") |> Enum.each(fn row ->

  # Extract data

end)

We iterate through each

Extract Column Data

Inside the loop, we get the column data:

[name_cell, group_cell, local_cell, img_cell] = Floki.find(row, "td, th")

name = Floki.text(name_cell) |> String.trim()
group = Floki.text(group_cell) |> String.trim()

local_name = Floki.text(local_cell) |> String.trim()

img = Floki.attribute(img_cell, "src")

We use text/1 for text and attribute/2 for attributes.

Download Images

To download and save images:

unless img == nil do
  {:ok, image} = HTTPoison.get(img, [], hackney: [pool: :default])

  File.write("dog_images/#{name}.jpg", image.body)
end

We reuse the HTTP client and write the image binary to a file.

Store Extracted Data

We store the extracted data:

names = [name | names]
groups = [group | groups]
local_names = [local_name | local_names]
images = [img | images]

The lists can then be processed as needed.

And that's it! Here is the full code:

# Imports
import HTTPoison
import Floki

# Lists to store data
names = []
groups = []
local_names = []
images = []

# Fetch HTML
url = "<https://commons.wikimedia.org/wiki/List_of_dog_breeds>"

response = HTTPoison.get!(url, [], hackney: [pool: :default, recv_timeout: 25000])

html = response.body |> Floki.parse_document!()

# Find table
table = html |> Floki.find("table.wikitable.sortable")

# Iterate rows
Floki.find(table, "tr") |> Enum.each(fn row ->

  [name_cell, group_cell, local_cell, img_cell] = Floki.find(row, "td, th")

  name = Floki.text(name_cell) |> String.trim()
  group = Floki.text(group_cell) |> String.trim()

  local_name = Floki.text(local_cell) |> String.trim()

  img = Floki.attribute(img_cell, "src")

  unless img == nil do
    {:ok, image} = HTTPoison.get(img, [], hackney: [pool: :default])

    File.write("dog_images/#{name}.jpg", image.body)
  end

  names = [name | names]
  groups = [group | groups]
  local_names = [local_name | local_names]
  images = [img | images]

end)

This provides a complete Elixir solution using HTTPoison and Floki to scrape data and images from HTML tables. The same approach can apply to many websites.

While these examples are great for learning, scraping production-level sites can pose challenges like CAPTCHAs, IP blocks, and bot detection. Rotating proxies and automated CAPTCHA solving can help.

Proxies API offers a simple API for rendering pages with built-in proxy rotation, CAPTCHA solving, and evasion of IP blocks. You can fetch rendered pages in any language without configuring browsers or proxies yourself.

This allows scraping at scale without headaches of IP blocks. Proxies API has a free tier to get started. Check out the API and sign up for an API key to supercharge your web scraping.

With the power of Proxies API combined with Python libraries like Beautiful Soup, you can scrape data at scale without getting blocked.