Building a Simple Proxy Rotator with Objective-C

In the early stages of a web crawling project or when you need to scale to just a few hundred requests, you may want a simple proxy rotator that populates itself from free proxy pools available on the internet.

We can use a website like https://sslproxies.org/ to fetch public proxies every few minutes and use them in our Objective-C projects.

This is what the site looks like:

And if you check the HTML using the inspect tool, you will see the full content is encapsulated in a table with the id proxylisttable

The IP and port are the first and second elements in each row.

We can use the following code to select the table and its rows to iterate on and further pull out the first and second elements of the elements.

Fetching the Proxy List

First, we'll fetch the HTML of the proxy list page using NSURLSession:

NSURL *url = [NSURL URLWithString:@"<https://sslproxies.org/>"];
NSURLSessionConfiguration *config = [NSURLSessionConfiguration defaultSessionConfiguration];
NSURLSession *session = [NSURLSession sessionWithConfiguration:config];

NSURLSessionDataTask *dataTask = [session dataTaskWithURL:url completionHandler:^(NSData * _Nullable data, NSURLResponse * _Nullable response, NSError * _Nullable error) {
  // Handle response
}];

[dataTask resume];

We should also set a custom user agent string to mimic a web browser:

NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:url];
[request setValue:@"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9" forHTTPHeaderField:@"User-Agent"];

Parsing the HTML

To parse the HTML, we can use a library like HPPLE. First import it:

#import <Hpple/Hpple.h>

Then we can parse the HTML data inside the completion handler:

TFHpple *xpathParser = [TFHpple hppleWithHTMLData:data];

NSString *xpathQuery = @"//table[@id='proxylisttable']/tr";
NSArray *rows = [xpathParser searchWithXPathQuery:xpathQuery];

for (TFHppleElement *row in rows) {
  // Extract IP and port
}

The IP address and port number are in the first and second elements of each , so we can extract them like:

NSString *ip = [[row firstChild] content];
NSString *port = [[row childAtIndex:1] content];

Storing and Selecting a Random Proxy

Let's store the proxies in an array and pick one randomly:

NSMutableArray *proxies = [NSMutableArray new];

// Inside parsing loop...

[proxies addObject:@{@"ip": ip, @"port": port}];

// Later to get a random proxy...

NSUInteger randomIndex = arc4random_uniform([proxies count]);
NSDictionary *randomProxy = proxies[randomIndex];
NSString *randomIP = randomProxy[@"ip"];
NSString *randomPort = randomProxy[@"port"];

Putting It All Together

// Fetch and parse proxies

NSURL *url = [NSURL URLWithString:@"<https://sslproxies.org/>"];

NSMutableURLRequest *request = [NSMutableURLRequest requestWithURL:url];
[request setValue:@"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9" forHTTPHeaderField:@"User-Agent"];

NSURLSession *session = [NSURLSession sharedSession];
NSURLSessionDataTask *dataTask = [session dataTaskWithRequest:request completionHandler:^(NSData *data, NSURLResponse *response, NSError *error) {

  NSMutableArray *proxies = [NSMutableArray new];

  TFHpple *xpathParser = [TFHpple hppleWithHTMLData:data];

  NSString *xpathQuery = @"//table[@id='proxylisttable']/tr";
  NSArray *rows = [xpathParser searchWithXPathQuery:xpathQuery];

  for (TFHppleElement *row in rows) {

    NSString *ip = [[row firstChild] content];
    NSString *port = [[row childAtIndex:1] content];

    [proxies addObject:@{@"ip": ip, @"port": port}];

  }

  // Select random proxy

  NSUInteger randomIndex = arc4random_uniform([proxies count]);
  NSDictionary *randomProxy = proxies[randomIndex];

  NSString *randomIP = randomProxy[@"ip"];
  NSString *randomPort = randomProxy[@"port"];

  // Use proxy...

}];

[dataTask resume];

This provides a simple way to fetch a list of proxies and select one randomly to use in your Objective-C application. Be sure to call it every few minutes to rotate proxies.

If you want to use this in production and want to scale to thousands of links, then you will find that many free proxies won't hold up under the speed and reliability requirements. In this scenario, using a rotating proxy service to rotate IPs is almost a must.

Otherwise, you tend to get IP blocked a lot by automatic location, usage, and bot detection algorithms.

Our rotating proxy server Proxies API provides a simple API that can solve all IP Blocking problems instantly.

With millions of high speed rotating proxies located all over the world • With our automatic IP rotation • With our automatic User-Agent-String rotation (which simulates requests from different, valid web browsers and web browser versions) • With our automatic CAPTCHA solving technology

Hundreds of our customers have successfully solved the headache of IP blocks with a simple API.

A simple API can access the whole thing like below in any programming language.

curl "<http://api.proxiesapi.com/?key=API_KEY&url=https://example.com>"

We have a running offer of 1000 API calls completely free. Register and get your free API Key here.

Building a Simple Proxy Rotator with Objective-C

Fetching the Proxy List

Parsing the HTML

Storing and Selecting a Random Proxy

Putting It All Together

Browse by tags:

Browse by language:

The easiest way to do Web Scraping

Building a Simple Proxy Rotator with Objective-C

Fetching the Proxy List

Parsing the HTML

Storing and Selecting a Random Proxy

Putting It All Together

The easiest way to do Web Scraping

Don't leave just yet!