If you've done any significant web scraping or bot development, chances are you've used Axios - the popular HTTP client for JavaScript. With its promise-based interface and easy configuration, Axios makes sending GET requests and posting data a breeze.
However, I learned the hard way that as your projects scale and make an exponentially growing number of requests, websites have ways of detecting and blocking unwanted traffic. After a few too many IP bans, I knew it was time to set up proxies.
In this guide, I want to share the proxy integration strategies I've picked up over years of web scraping and bot building with Axios. Consider it your crash course in using proxies with Axios - with hard-won tips so you can avoid my past pitfalls!
Why Axios? Why Proxies?
First - a quick primer on Axios and proxies for those unfamiliar.
Axios burst onto the scene a few years back as an alternative to the Fetch API for making HTTP calls from JavaScript. It gained popularity because of its ease of use, promise-based interface, and ability to transform JSON automatically.
Here was a simple GET request with Axios to grab some data:
const axios = require('axios');
axios.get('<https://api.example.com/data>')
.then(response => {
// Display response data
console.log(response.data);
});
However, I soon realized that as my scrapers and bots started hammering websites with thousands of requests an hour, I'd get ominous 403 Forbidden errors or find myself blocked completely!
This is where proxies came in handy. Proxies act as an intermediary between your client and target website. Rather than coming directly from your IP address, requests go through the proxy IP. This masks who's making the requests.
Many large sites block traffic from certain regions too. Proxies enabled me to appear from a whole range of geographic locations.
Now that you've got the 30,000 foot view on Axios and proxies, let's jump into implementation!
Setting Up Axios
Since proxies build on top of standard Axios functionality, we need to cover some Axios basics first.
The first step is installing Axios using npm or yarn:
npm install axios
Then require Axios at the top of your script:
const axios = require('axios');
With the module imported, we can start making requests. Let's grab some placeholder data from JSONPlaceholder:
axios.get('<https://jsonplaceholder.typicode.com/todos/1>')
.then(response => {
console.log(response.data);
})
.catch(error => {
console.log(error);
});
Run this and you should see data for the first todo item logged. Nice start!
Now let's move on to the main event - integrating proxies.
Using HTTP/HTTPS Proxies
Many scrapped sites look for repeated requests coming from the same IP address. To prevent blocks, you'll want to route requests through proxy servers to mask your originating IP.
Axios supports HTTP and HTTPS proxies directly using a simple configuration dictionary.
It should resemble:
proxy: {
protocol: 'http',
host: 'XXX.XXX.XXX', // Proxy IP
port: 8080 // Proxy port
}
You would then plug this proxy config into individual Axios requests like:
const proxyConfig = {
// Proxy settings
};
axios.get('<https://api.example.com>', {
proxy: proxyConfig
});
Or set
const axios = require('axios');
const proxyConfig = {
// Proxy settings
};
const api = axios.create({
proxy: proxyConfig
});
api.get('/endpoing'); // Uses proxy
This was life-changing when I first got it working! By alternating just a few residential proxies, my scrapers were no longer getting blocked on large sites.
Pro Tip: Pay attention to using the proper protocol in your proxy config - HTTP or HTTPS. I spent longer than I'd like to admit scratching my head after mindlessly copying an HTTP proxy 😅
Dealing with SOCKS Proxies
When I started using proxies with rotating IP addresses, many were SOCKS proxies rather than plain old HTTP.
Axios itself doesn't directly work with SOCKS out of the box unfortunately. But thankfully there are libraries like socks-proxy-agent that make integration simple.
Here was the code to use socks-proxy-agent with Axios:
// Import SocksProxyAgent library
const SocksProxyAgent = require('socks-proxy-agent');
// Create agent with SOCKS proxy settings
const proxyAgent = new SocksProxyAgent(`socks://user:[email protected]:8080`);
// Use the agent in Axios requests
axios.get('<https://api.example.com>', {
httpAgent: proxyAgent,
httpsAgent: proxyAgent
});
While slightly more verbose, this unlocked all sorts of new proxy opportunities once I figured it out!
Advanced Proxy Handling
Let's level up our proxy usage...
Authentication
Many premium proxies require credentials to authenticate access.
Axios allows specifying these conveniently in the proxy config versus having to embed them in the proxy URL:
proxy: {
host: '1.2.3.4',
port: 8080,
auth: {
username: 'my_user',
password: 'top$ecret'
}
}
I setup all my proxy credentials this way - keeps things clean!
Environment Variables
Hard-coding your proxy particulars isn't ideal. I recommend using environment variables instead:
HTTP_PROXY=http://username:[email protected]:8080
HTTPS_PROXY=http://username:[email protected]:8080
Axios automatically picks these up at runtime. Just set
Proxy Rotation
To distribute load and avoid blocks, you'll want to rotate across multiple proxies randomly.
Here was my simple proxy rotation function:
// Array of proxy objects
const proxies = [];
// Get random proxy
const getProxy = () => {
return proxies[~~(Math.random() * proxies.length)];
}
// Use random proxy in requests
axios.get('<https://api.example.com>', {
proxy: getProxy()
});
It chooses a proxy randomly on each call, spreading requests evenly across IPs.
While basic, this method served me well on many small to medium scrapers!
Example: Retail Pricing Bot
Enough theory - let's apply some of these proxies concepts to a practical example.
Say we want to monitor prices changes across products on an retail site like Amazon or BestBuy. Running scripts from a single IP that hits their server continuously is a quick path to getting blocked.
Here is how we could build a pricing bot using proxies:
const axios = require('axios');
const jsdom = require('jsdom'); // Parses HTML
// Array of HTTP proxies to rotate across
const proxies = [];
// Grab pricing page of product
async function getPrice(url) {
// Select random proxy
const proxy = proxies[~~(Math.random() * proxies.length)];
try {
// Fetch page HTML through proxy
const { data } = await axios.get(url, {
proxy
});
// Parse HTML to find pricing
const dom = new JSDOM(data);
const price = dom.window.document.querySelector('.price').innerHTML;
return price;
} catch(err) {
console.log('Proxy Connection Error', err);
}
}
module.exports = { getPrice };
Now we can import getPrice to any app that needs to monitor retail pricing data without fear of blocks!
While basic, this shows how proxies enable you to gather data that would otherwise be unavailable.
Key Takeaways
After years using Axios for large scale web scraping and bot development, here are vital things I've learned when using proxies:
And perhaps most importantly, premium residential proxies are 100% worth the modest investment. The reliability, rotating IPs, and reduced blocks will repay themselves in no time from the extra data you can access.
I hope walking through real-world examples and lessons from the trenches gives you a solid foundation for integrating proxies with Axios. No more pulling code snippets blindly from blogs!
Happy proxying and scraping! Speak soon 👋
Frequently Asked Questions
Q: What are some alternatives to Axios for making proxy requests in JavaScript?
Two great options are node-fetch which offers a Promise-based interface like Axios, and Got which builds on node-fetch adding some convenient utilities and features.
Q: Can proxies be used in frontend JavaScript code?
Unfortunately frontend JavaScript code (client-side code running in the browser) can't directly connect through proxies. Proxies become relevant for server-side apps and scrapers written with Node.js.
Q: Does Axios work well with mobile proxies?
Absolutely! Axios doesn't care or handle proxies any differently based on type. You configure static residential, datacenter, and mobile proxies the exact same way shown in this article.
Q: Why do my proxies stop working suddenly? They were working fine yesterday!
I feel your pain - I once had a big scraper grind to a halt when a handful of IPs suddenly got blocked by the target domain overnight! This unpredictability is the downside of using free public proxies unfortunately.
The lesson I learned was to use providers offering reliable premium residential proxies. The cost is low compared to the benefit of stable IPs tested against anti-scraping defenses.