get specific data from web page

Question

0.00/5 (No votes)

See more:

Hi guys,
I have to get data of thousands of companies from a website and I have to write a program to make it automatically. I searhced it on google but couldn't do it.

First of all, I enter the web page which has city name links. When I click on the city name, it send me a new page. There are company name links at this page. When I click on the company name, it sends me a new page. Then, I have to search that page and get the related information(company name,phone number, address etc.) into an excel worksheet. Can you help me about how I can do it, please?

Posted 16-Nov-11 22:42pm

Çağlar Tolga Tetik

Add a Solution

2 solutions

Solution 1

You need to be able to find the appropriate HTML tag for the information you want by parsing the HtmlElement[^] or its collection in the Web document. Once you have found the right element then you can extract the individual fields of data, assuming they are all tagged in the way you want. For more information on these techniques try a search on "Screenscraper" in Google or the CodeProject articles.

Posted 16-Nov-11 23:48pm

Richard MacCutchan

Comments

Sergey Alexandrovich Kryukov 26-Dec-11 14:47pm

My 5. I did not notice that I added my answer on next day after this one, but it's different.
--SA

Richard MacCutchan 27-Dec-11 6:10am

As usual your answer is informative and comprehensive; 5 for that.

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Sergey Alexandrovich Kryukov · Accepted Answer · 2011-11-17T21:52:00

This is called Web Scraping, http://en.wikipedia.org/wiki/Web_scraping[^].

You need to use the class System.Net.HttpWebRequest, but to create one, your will need a variable of compile-time class System.Net.WebRequest, see:
http://msdn.microsoft.com/en-us/library/system.net.webrequest.aspx[^] (HttpWebRequest sample here),
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx[^].

Using HttpWebRequest you will obtain a document. If this is a HTML document, you will need to parse it. It would be good if this is a document well-formed as XML, then you could parse it using one of .NET XML parsers. Not all Web pages are like that, unfortunately, so you may need HTML parser which does not require well-formed XML compliance. Try this one: http://www.majestic12.co.uk/projects/html_parser.php[^].

—SA