Click here to Skip to main content
12,887,373 members (30,847 online)
Rate this:
Please Sign up or sign in to vote.
See more: C#
Hi guys,
I have to get data of thousands of companies from a website and I have to write a program to make it automatically. I searhced it on google but couldn't do it.

First of all, I enter the web page which has city name links. When I click on the city name, it send me a new page. There are company name links at this page. When I click on the company name, it sends me a new page. Then, I have to search that page and get the related information(company name,phone number, address etc.) into an excel worksheet. Can you help me about how I can do it, please?
Posted 16-Nov-11 22:42pm
Rate this: bad
Please Sign up or sign in to vote.

Solution 1

You need to be able to find the appropriate HTML tag for the information you want by parsing the HtmlElement[^] or its collection in the Web document. Once you have found the right element then you can extract the individual fields of data, assuming they are all tagged in the way you want. For more information on these techniques try a search on "Screenscraper" in Google or the CodeProject articles.
SAKryukov 26-Dec-11 14:47pm
My 5. I did not notice that I added my answer on next day after this one, but it's different.
Richard MacCutchan 27-Dec-11 6:10am
As usual your answer is informative and comprehensive; 5 for that.
Rate this: bad
Please Sign up or sign in to vote.

Solution 2

This is called Web Scraping,[^].

You need to use the class System.Net.HttpWebRequest, but to create one, your will need a variable of compile-time class System.Net.WebRequest, see:[^] (HttpWebRequest sample here),[^].

Using HttpWebRequest you will obtain a document. If this is a HTML document, you will need to parse it. It would be good if this is a document well-formed as XML, then you could parse it using one of .NET XML parsers. Not all Web pages are like that, unfortunately, so you may need HTML parser which does not require well-formed XML compliance. Try this one:[^].

Mika Wendelius 26-Dec-11 14:35pm
Very informative, 5.
SAKryukov 26-Dec-11 14:46pm
Thank you, Mika.
Mohamed Mitwalli 31-Jul-12 2:29am
Sergey Alexandrovich Kryukov 31-Jul-12 11:15am
Thank you, Mohamed.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

    Print Answers RSS
Top Experts
Last 24hrsThis month

Advertise | Privacy | Mobile
Web02 | 2.8.170424.1 | Last Updated 18 Nov 2011
Copyright © CodeProject, 1999-2017
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100