Web Scraping Wikipedia in Scala

Wikipedia is an incredibly useful source of structured data on almost any topic imaginable. Much of this data is presented in table format, making it perfect for scraping. Web scraping is the process of programmatically extracting data from websites. It allows you to harvest large amounts of data without tedious manual copying and pasting.

In this beginner's guide, we'll walk through a simple example of how to scrape Wikipedia using Scala and an HTML parsing library called Jsoup. Our goal will be to gather key data on all US presidents from Wikipedia and print it out.

This is the table we are talking about

The Key Steps

Here are the main steps we'll walk through:

1. Importing libraries
2. Defining the URL
3. Setting the user agent string
4. Sending the HTTP request
5. Parsing the HTML
6. Extracting the data
7. Printing the scraped data

It may sound daunting right now, but web scraping is accessible even to coding beginners with the right guidance. Let's get started!

1. Importing Libraries

We first need to import the libraries that we'll use for scraping and parsing:

import org.jsoup.Jsoup

import scala.collection.JavaConverters._

Jsoup handles connecting to web pages and parsing HTML. The JavaConverters package helps convert between Java and Scala collections since Jsoup uses Java collections under the hood.

2. Defining the URL

Next, we store the Wikipedia URL containing president data in a variable:

val url = "<https://en.wikipedia.org/wiki/List_of_presidents_of_the_United_States>"

Hard-coding the URL like this is OK for one-off scraping. For larger projects, consider loading URLs from a file or database.

3. Setting the User Agent

Websites can detect and block scraping bots. To mimic a real browser, we need to set a valid user agent header:

val userAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

This makes Wikipedia think we're Chrome running on Windows 10. User agents can be tricky to get right - we used a real-world Chrome one above.

4. Sending the HTTP Request

Now we can send an HTTP GET request for the URL. We pass the user agent in:

val doc = Jsoup.connect(url).userAgent(userAgent).get()

The get() sends the request and fetches the page HTML. This is saved in the doc variable for parsing next.

5. Parsing the HTML

Jsoup represents HTML documents as a nested Document Object Model (DOM) that we can query with CSS selectors, like jQuery.

Inspecting the page

When we inspect the page we can see that the table has a class called wikitable and sortable

Let's grab the presidents table:

val table = doc.select("table.wikitable.sortable").first()

This finds the

💡 Tip: Be as specific as possible with selectors to avoid grabbing unnecessary data.

6. Extracting the Data

Now we can iterate through the table rows, get each cell, and extract the text:

for (row <- table.select("tr").asScala.drop(1)) {

  val columns = row.select("td,th").asScala.toList

  val row_data = columns.map(_.text())

  data += row_data
}

This skips the header row, gets the text from every cell in each subsequent row, and stores it in our data list for later use.

7. Printing the Scrapped Data

Finally, we can neatly print out all president data:

for (president_data <- data) {

  println("Name: " + president_data(2))
  println("Term: " + president_data(3))

  // print more attributes...

}

And we have successfully scraped structured data from a Wikipedia table!

The full runnable code example is listed again below for reference:

import org.jsoup.Jsoup
import scala.collection.JavaConverters._

object WikipediaScraper {
  def main(args: Array[String]): Unit = {
    // Define the URL of the Wikipedia page
    val url = "https://en.wikipedia.org/wiki/List_of_presidents_of_the_United_States"

    // Define a user-agent header to simulate a browser request
    val userAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"

    // Send an HTTP GET request to the URL with the user-agent header
    val doc = Jsoup.connect(url).userAgent(userAgent).get()

    // Find the table with the specified class name
    val table = doc.select("table.wikitable.sortable").first()

    // Initialize empty lists to store the table data
    val data = scala.collection.mutable.ListBuffer.empty[List[String]]

    // Iterate through the rows of the table
    for (row <- table.select("tr").asScala.drop(1)) { // Skip the header row
      val columns = row.select("td,th").asScala.toList

      // Extract data from each column and append it to the data list
      val row_data = columns.map(_.text())
      data += row_data
    }

    // Print the scraped data for all presidents
    for (president_data <- data) {
      println("President Data:")
      println("Number:", president_data(0))
      println("Name:", president_data(2))
      println("Term:", president_data(3))
      println("Party:", president_data(5))
      println("Election:", president_data(6))
      println("Vice President:", president_data(7))
      println()
    }
  }
}

Key Takeaways

The key concepts we learned:

With just these basics, you can start scraping many useful sites. To level up:

In more advanced implementations you will need to even rotate the User-Agent string so the website cant tell its the same browser!

If we get a little bit more advanced, you will realize that the server can simply block your IP ignoring all your other tricks. This is a bummer and this is where most web crawling projects fail.

Overcoming IP Blocks

Investing in a private rotating proxy service like Proxies API can most of the time make the difference between a successful and headache-free web scraping project which gets the job done consistently and one that never really works.

Plus with the 1000 free API calls running an offer, you have almost nothing to lose by using our rotating proxy and comparing notes. It only takes one line of integration to its hardly disruptive.

Our rotating proxy server Proxies API provides a simple API that can solve all IP Blocking problems instantly.

Hundreds of our customers have successfully solved the headache of IP blocks with a simple API.

The whole thing can be accessed by a simple API like below in any programming language.

curl "http://api.proxiesapi.com/?key=API_KEY&url=https://example.com"

We have a running offer of 1000 API calls completely free. Register and get your free API Key here.