Avoid IP blocking with web scraping

Question

1.33/5 (3 votes)

See more:

Hi all,

I developed a web scraper (using C#) that should be able to make thousands of requests each time.

The problem is that the website's server will block my IP after a number of requests.

Questions:

1- How to prevent being blocked?
2- How to know when will the website's server will block my IP? I mean how to know my limit whether being certain amount of traffic or certain number of requests.

Thanks.

Posted 10-May-11 1:31am

Yas_EG

Updated 10-May-11 1:38am

v2

Add a Solution

3 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Nish Nishant · Answer 1 · 2011-05-10T01:50:00

There's no way to do this. The one way would to be to limit the scraping to a very slow rate which kinda nullifies the very purpose of scraping.

Alternatively, spread the scraping out to multiple domains. For example pick a 100 domains, get 1 page from domain-1, then the next from domain-2, and so on till domain-100, then get the 2nd page from domain-1, then from domain-2, and so on. The trick here is that this artificially slows down your scraping to 1/100 of its former speed (from the server's perspective), but you don't actually lose out on your scraping speeds because you are scraping from multiple sites. Makes sense?

CDP1802 · Answer 2 · 2011-05-10T01:44:00

1) That's very easy. Don't do anything that might lead to being considered a threat. Bombarding a server with 'thousands of requests each time' could be considered to be hostile.

2) Now that's a good idea. Each server should post information on how much abuse its owner will tolerate. Seriously, it's very much like when you misbehave at somebody's house. When the owner grabs you by the collar and shows you the door, then you know how far you could go.

GuneyGuney · Answer 3 · 2012-02-09T00:28:00

Solution 3

It's not possible to get block if you are able to use my way.

Use an ADSL modem like Airties, let your server use that internet connection and send reset command in a schedule.

That works like a charm. :)

Posted 9-Feb-12 0:28am

GuneyGuney

Avoid IP blocking with web scraping

3 solutions

Solution 2

Solution 1

Solution 3

Add your solution here

Preview 0