Data Scraping scraping

Question

0.00/5 (No votes)

See more:

Sir i want to ask how can one view pages of a website which has a check on number of pages to be viewed.

for example
http://maps.google.com/

allows you to view only 15pages but there are still more pages to be viewed

similarly
www.411.ca
there you can view only 20pages but there are still more pages

can someone give me some idea how can we view/scrap the above problem

Thanks

Posted 28-May-11 19:00pm

cocksure77

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Kim Togo · Accepted Answer · 2011-05-28T19:40:00

Solution 1

Slow the process done on scraping. If you do it to fast (faster then a human browsing the page), websites like google etc. can limit or throttle the amounts of request.

Posted 28-May-11 19:40pm

Kim Togo

Comments

[no name] 29-May-11 2:07am

sir just see the link

http://www.411.ca/search/business/page/1

when we manual see it contains more data than

http://www.411.ca/search/business/page/20

how can we view manualy or throgh software

http://www.411.ca/search/business/page/21

Kim Togo 29-May-11 2:16am

I do not know how you download the content and analyze it in your program. But when it comes to HTML, Javascript etc. The Internet Explorer, Google Chrome etc. processes data and show it to the user.

It think a better way is find out if there is a API that can be used like Web Service etc.