We at Proxies API have clients who have used our Rotating Proxies Service for all sorts of web crawling use cases.

Since we are a Rotating Proxy Service that developers use to scale their web crawling, our ideal audience is developers. So we are always on the lookout for opportunities to help developers—especially developers who are involved in web scraping web crawling projects.

So we need to know when someone asks a question about anything to do with Web crawling/Web Scraping or Data Mining technologies.

So we have built our little web crawler behind the scenes to Quora and other social networks like Twitter, Facebook, Linkedin, Google News for people needing help with these technologies.

Since we started doing this, this is one of our primary drivers of traffic, which happened by merely finding and answering questions online about these topics on time.

Since we started this internal crawler, we have detected about 500 places across Quora, Linkedin, Facebook, Twitter, Reddit, etc. where we can add value.

Our bot returns a database of these opportunities like below for us daily, and its fun to watch everyone on the team try to take a crack at answering/helping them.

It has prompted many interesting discussions and knowledge sharing and research sessions amongst the team.

We have already posted about 110 answers in various places in a systematic way. Sometimes, these questions inspire an in-depth blog post on our blog as well. We even got some people from Quora who directly reached out and became our customers, plus we are helping them set up their entire crawling infrastructure!

We also feel like we have a pulse on the pain points that our potential customers face and have grown to have real empathy for them by just actively looking to help in any way we can.