web scraping a gridview with paging

Question

0.00/5 (No votes)

See more:

I have some problems, I must screen scrape a web page that is an aspx-site where the interesting data is the text in a grid view. I have no problem to get the text from the first page in the grid view.

But how do I get the text from the second page, the third page etc.

The problem is that every page in the grid view has not a specific URL, instead it is a java script that is executing when I click on the paging link button.

For example the java script for a paging java script is:
javascript:__doPostBack('ctl00$cphmaincontent$lbntNavigate2,3..up last page','')

How do I simulate this?

Posted 26-Jan-14 19:58pm

Member 10500506

Updated 26-Jan-14 19:59pm

v2

Add a Solution

Comments

Sergey Alexandrovich Kryukov 27-Jan-14 2:11am

Not everything could be scraped; for certain pages, the whole problem cannot even be formulated...
—SA

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sergey Alexandrovich Kryukov · Answer 1 · 2014-01-26T20:17:00

Please see my comment to the question… However, if this is paging which behaves in some regular way, you can set a HTTP spy and see what HTTP requests are sent on each paging event. Apparently, you cannot assume any particular server-side technology for the site being scrapped. So, you need to learn how it works and mimic the behavior of the client site.

One method I used is some HTTP spy application. I, for example, use the one created as a plug-in to SeaMonkey/Firefox, its name is Http Fox, but I know for sure that there are similar tools, for different browsers and stand-along one. Using such tool, you can pretty easy find out what's going on. Besides, all source Javascript code is always readable to you, you can study it.

I want to emphasize that nothing can guarantee 100% success for all sites. However, you will probably find out very general most typically used classes of cases. For example, most ASP.NET pages with grid view use pretty much the same paging mechanism.

—SA

Mitchell J. · Answer 2 · 2014-01-26T20:08:00

Solution 1

If you are using a WebBrowserControl to load the pages so you can scrape them, then you can try invoking a click event on the paging button, like this:

C#

webbrowser.Document.GetElementById("PagingButtonID").InvokeMember("click");

Posted 26-Jan-14 20:08pm

Mitchell J.