Have you ever wanted to analyze real estate listing data from sites like Realtor.com? Web scraping using tools like Jsoup provide a powerful way to automatically extract that data for custom analysis.

This is the listings page we are talking about…

In this comprehensive guide for beginners, you'll learn step-by-step how to use Jsoup to scrape key details from Realtor listings in Kotlin. Follow along to get hands-on practice with core scraping concepts like:

We'll go extremely in-depth on the most complex part - the CSS selectors - since that's what allows extracting the right data points.

By the end, you'll understand how each piece of this scraper works to extract details like:

...and more from any result on a Realtor.com search page!

The code can work as-is to scrape listings for any city, while what you learn will apply to building scrapers for virtually any site.

Let's dig in!

Scraping Overview

First, what exactly happens in this program?

At a high level, it:

1. Sends a GET request to retrieve the HTML from a Realtor search URL
2. Parses the HTML
3. Uses CSS selectors to pinpoint specific elements
4. Extracts text from those elements 5.Outputs the extracted details

That's web scraping in a nutshell! It allows gathering structured data from sites even if they don't have public APIs.

Now let's break down the process for this specific scraper...

Importing Jsoup

Jsoup handles most of the heavy lifting for us. To start, we import it:

import org.jsoup.Jsoup
import org.jsoup.nodes.Document
import org.jsoup.nodes.Element

This gives access to all the scraping functionality.

We'll see examples later of how these work together.

Defining the Target URL

The first step is choosing what page to scrape. In this case, we want results from a Realtor search:

val url = "<https://www.realtor.com/realestateandhomes-search/San-Francisco_CA>"

This URL will contain many individual property listings we want to extract details from.

Good to Know Tip: You can tweak the location in the URL to scrape other cities!

Creating a User Agent Header

Websites detect programmatic access like scrapers by looking for missing browser characteristics like user agent strings.

To mimic a real browser, we pass a user agent header:

val userAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"

This spoofed string matches Chrome on Windows 10.

Pro Tip: I recommend services like https://www.whatismybrowser.com to get your actual browser's user agent string for scraping to better blend in!

Fetching the Page HTML

With the target URL and user agent defined, Jsoup can now grab the page HTML:

val doc = Jsoup.connect(url).userAgent(userAgent).get()

Breaking this down:

The returned doc contains the entire source code of the page. Think of it like an object representation of the raw HTML.

We can check if (doc != null) to confirm it loaded properly before scraping.

Lightbulb Moment: The user agent header tricks Realtor.com into thinking the request came from a real browser!

Selecting Elements to Scrape

Here's where the real magic happens - using CSS selectors to pinpoint parts of the page to extract.

Understanding CSS Selectors

CSS selectors allow targeting HTML elements by their id, class, tag name, attributes, hierarchy in the DOM tree and more.

Some examples:

/* Target element with id property matching "myId" */
#myId

/* Target elements with class name matching "myClass" */
.myClass

/* Target h1 elements */
h1

/* Target elements with attribute name="value" */
[name="value"]

Jsoup translates these selectors into matching Java Element objects from the HTML.

The full syntax and possibilities get complex, but as you'll see, just a few selector types cover most scraping cases!

Selecting Realtor Listing Blocks

Inspecting the element

When we inspect element in Chrome we can see that each of the listing blocks is wrapped in a div with a class value as shown below…

Now back to our program...

The first thing we want is each overall listing block from the page. Realtor conveniently marks these with a class name:

val listingBlocks = doc.select("div.BasePropertyCard\\_propertyCardWrap\\_\\_J0xUj")

Breaking this selector down:

This matches all

val brokerInfo = listingBlock.selectFirst("div.BrokerTitle\\_brokerTitle\\_\\_ZkbBW")

val brokerName = brokerInfo?.selectFirst("span.BrokerTitle\\_titleText\\_\\_20u1P")?.text()?.trim() ?: "N/A"

val status = listingBlock.selectFirst("div.message")?.text()?.trim() ?: "N/A"

val price = listingBlock.selectFirst("div.card-price")?.text()?.trim() ?: "N/A"

val bedsElement = listingBlock.selectFirst("li\\[data-testid=property-meta-beds\\]")
// ...get text from bedsElement

println("Broker: $brokerName")
println("Status: $status")
println("Price: $price")
// ...

import org.jsoup.Jsoup
import org.jsoup.nodes.Document
import org.jsoup.nodes.Element

fun main() {
    // Define the URL of the Realtor.com search page
    val url = "https://www.realtor.com/realestateandhomes-search/San-Francisco_CA"

    // Define a User-Agent header
    val userAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"

    // Send a GET request to the URL with the User-Agent header
    val doc = Jsoup.connect(url).userAgent(userAgent).get()

    // Check if the request was successful (status code 200)
    if (doc != null) {
        // Find all the listing blocks using the provided class name
        val listingBlocks = doc.select("div.BasePropertyCard_propertyCardWrap__J0xUj")

        // Loop through each listing block and extract information
        for (listingBlock in listingBlocks) {
            // Extract the broker information
            val brokerInfo = listingBlock.selectFirst("div.BrokerTitle_brokerTitle__ZkbBW")
            val brokerName = brokerInfo?.selectFirst("span.BrokerTitle_titleText__20u1P")?.text()?.trim() ?: "N/A"

            // Extract the status (e.g., For Sale)
            val status = listingBlock.selectFirst("div.message")?.text()?.trim() ?: "N/A"

            // Extract the price
            val price = listingBlock.selectFirst("div.card-price")?.text()?.trim() ?: "N/A"

            // Extract other details like beds, baths, sqft, and lot size
            val bedsElement = listingBlock.selectFirst("li[data-testid=property-meta-beds]")
            val bathsElement = listingBlock.selectFirst("li[data-testid=property-meta-baths]")
            val sqftElement = listingBlock.selectFirst("li[data-testid=property-meta-sqft]")
            val lotSizeElement = listingBlock.selectFirst("li[data-testid=property-meta-lot-size]")

            // Check if the elements exist before extracting their text
            val beds = bedsElement?.text()?.trim() ?: "N/A"
            val baths = bathsElement?.text()?.trim() ?: "N/A"
            val sqft = sqftElement?.text()?.trim() ?: "N/A"
            val lotSize = lotSizeElement?.text()?.trim() ?: "N/A"

            // Extract the address
            val address = listingBlock.selectFirst("div.card-address")?.text()?.trim() ?: "N/A"

            // Print the extracted information
            println("Broker: $brokerName")
            println("Status: $status")
            println("Price: $price")
            println("Beds: $beds")
            println("Baths: $baths")
            println("Sqft: $sqft")
            println("Lot Size: $lotSize")
            println("Address: $address")
            println("-".repeat(50))  // Separating listings
        }
    } else {
        println("Failed to retrieve the page.")
    }
}