Click here to Skip to main content
Rate this: bad
Please Sign up or sign in to vote.
See more: C#
Hi guys,
I have to get data of thousands of companies from a website and I have to write a program to make it automatically. I searhced it on google but couldn't do it.
First of all, I enter the web page which has city name links. When I click on the city name, it send me a new page. There are company name links at this page. When I click on the company name, it sends me a new page. Then, I have to search that page and get the related information(company name,phone number, address etc.) into an excel worksheet. Can you help me about how I can do it, please?
Posted 16-Nov-11 23:42pm
Rate this: bad
Please Sign up or sign in to vote.

Solution 1

You need to be able to find the appropriate HTML tag for the information you want by parsing the HtmlElement[^] or its collection in the Web document. Once you have found the right element then you can extract the individual fields of data, assuming they are all tagged in the way you want. For more information on these techniques try a search on "Screenscraper" in Google or the CodeProject articles.
SAKryukov at 26-Dec-11 14:47pm
My 5. I did not notice that I added my answer on next day after this one, but it's different.
Richard MacCutchan at 27-Dec-11 6:10am
As usual your answer is informative and comprehensive; 5 for that.
Rate this: bad
Please Sign up or sign in to vote.

Solution 2

This is called Web Scraping,[^].
You need to use the class System.Net.HttpWebRequest, but to create one, your will need a variable of compile-time class System.Net.WebRequest, see:[^] (HttpWebRequest sample here),[^].
Using HttpWebRequest you will obtain a document. If this is a HTML document, you will need to parse it. It would be good if this is a document well-formed as XML, then you could parse it using one of .NET XML parsers. Not all Web pages are like that, unfortunately, so you may need HTML parser which does not require well-formed XML compliance. Try this one:[^].
Mika Wendelius at 26-Dec-11 14:35pm
Very informative, 5.
SAKryukov at 26-Dec-11 14:46pm
Thank you, Mika.
Mohamed Mitwalli at 31-Jul-12 2:29am
Sergey Alexandrovich Kryukov at 31-Jul-12 11:15am
Thank you, Mohamed.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
0 Peter Leow 160
1 Mika Wendelius 156
2 Abhinav S 155
3 Sergey Alexandrovich Kryukov 115
4 TheRealSteveJudge 114
0 Sergey Alexandrovich Kryukov 8,593
1 OriginalGriff 6,536
2 Peter Leow 3,727
3 Zoltán Zörgő 3,436
4 Richard MacCutchan 2,417

Advertise | Privacy | Mobile
Web01 | 2.8.150123.1 | Last Updated 18 Nov 2011
Copyright © CodeProject, 1999-2015
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100