Web Scraping with Go & ChatGPT

Go is a great language for web scraping thanks to its speed, concurrency support and simple syntax. ChatGPT is an AI assistant that can provide explanations and generate code for web scraping tasks. This article covers web scraping in Go with ChatGPT's assistance.

Setting Up Go for Web Scraping

You'll need Go installed along with these packages:

// net/http for requests

// goquery for DOM parsing
go get github.com/PuerkitoBio/goquery

// csv for CSV output
go get github.com/gocarina/gocsv

Introduction to Web Scraping in Go

Web scraping involves sending HTTP requests to websites and extracting data from the HTML, JSON or XML responses. Useful Go packages:

Typical web scraping workflow:

Using ChatGPT for Web Scraping Help

ChatGPT is an AI assistant created by OpenAI to be helpful, harmless, and honest. It can provide explanations and generate code snippets for web scraping:

Getting Explanations

Ask ChatGPT to explain web scraping concepts or specifics:

Generating Code Snippets

Give a description of what you want to scrape and have ChatGPT provide starter Go code:

Validate any code before using.

Improving Prompts

Ask ChatGPT to suggest ways to improve your prompt if it doesn't provide helpful responses.

Asking Follow-up Questions

Chat with ChatGPT to get explanations for any other questions you have.

Explaining Errors

Share any errors and ask ChatGPT to debug and explain the problem.

Web Scraping Example Using ChatGPT

Let's walk through scraping a Wikipedia page with ChatGPT's help.

Goal

Extract the chronology table from: https://en.wikipedia.org/wiki/Chronology_of_the_universe

Step 1: Download page

ChatGPT: Go code to download this page:
<https://en.wikipedia.org/wiki/Chronology_of_the_universe>

// ChatGPT provides this code

import "net/http"

resp, err := http.Get("<https://en.wikipedia.org/wiki/Chronology_of_the_universe>")
if err != nil {
  // handle error
}
defer resp.Body.Close()

body, err := ioutil.ReadAll(resp.Body)

Step 2: Inspect HTML, table has class wikitable

Step 3: Extract table data to CSV

ChatGPT: Go code to extract wikitable table to CSV

// ChatGPT provides this code

import (
  "github.com/PuerkitoBio/goquery"
  "github.com/gocarina/gocsv"
)

doc, err := goquery.NewDocumentFromReader(bytes.NewReader(body))

table := doc.Find("table.wikitable").First()

var records [][]string

table.Find("tr").Each(func(i int, s *goquery.Selection){

  var row []string

  s.Find("th, td").Each(func(j int, s2 *goquery.Selection){
    row = append(row, s2.Text())
  })

  records = append(records, row)

})

// write records to CSV
// ...

This shows using ChatGPT to get Go scraping code fast.

Conclusion

Key points:

ChatGPT + Go is great for creating web scrapers.

However, some limitations:

A more robust solution is using a web scraping API like Proxies API

Proxies API provides:

Easily scrape any site:

resp, err := http.Get("<https://api.proxiesapi.com/?url=example.com&key=XXX>")

Get started now with 1000 free API calls to supercharge your web scraping!