Click here to Skip to main content
Rate this: bad
good
Please Sign up or sign in to vote.
See more: C#
Hi guys,
I have to get data of thousands of companies from a website and I have to write a program to make it automatically. I searhced it on google but couldn't do it.
 
First of all, I enter the web page which has city name links. When I click on the city name, it send me a new page. There are company name links at this page. When I click on the company name, it sends me a new page. Then, I have to search that page and get the related information(company name,phone number, address etc.) into an excel worksheet. Can you help me about how I can do it, please?
Posted 16-Nov-11 22:42pm
Rate this: bad
good
Please Sign up or sign in to vote.

Solution 1

You need to be able to find the appropriate HTML tag for the information you want by parsing the HtmlElement[^] or its collection in the Web document. Once you have found the right element then you can extract the individual fields of data, assuming they are all tagged in the way you want. For more information on these techniques try a search on "Screenscraper" in Google or the CodeProject articles.
  Permalink  
Comments
SAKryukov at 26-Dec-11 14:47pm
   
My 5. I did not notice that I added my answer on next day after this one, but it's different. --SA
Richard MacCutchan at 27-Dec-11 6:10am
   
As usual your answer is informative and comprehensive; 5 for that.
Rate this: bad
good
Please Sign up or sign in to vote.

Solution 2

This is called Web Scraping, http://en.wikipedia.org/wiki/Web_scraping[^].
 
You need to use the class System.Net.HttpWebRequest, but to create one, your will need a variable of compile-time class System.Net.WebRequest, see:
http://msdn.microsoft.com/en-us/library/system.net.webrequest.aspx[^] (HttpWebRequest sample here),
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx[^].
 
Using HttpWebRequest you will obtain a document. If this is a HTML document, you will need to parse it. It would be good if this is a document well-formed as XML, then you could parse it using one of .NET XML parsers. Not all Web pages are like that, unfortunately, so you may need HTML parser which does not require well-formed XML compliance. Try this one: http://www.majestic12.co.uk/projects/html_parser.php[^].
 
—SA
  Permalink  
Comments
Mika Wendelius at 26-Dec-11 14:35pm
   
Very informative, 5.
SAKryukov at 26-Dec-11 14:46pm
   
Thank you, Mika. --SA
Mohamed Mitwalli at 31-Jul-12 2:29am
   
5+
Sergey Alexandrovich Kryukov at 31-Jul-12 11:15am
   
Thank you, Mohamed. --SA

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
Your Filters
Interested
Ignored
     
0 Shai Vashdi 1,448
1 Tadit Dash 270
2 OriginalGriff 223
3 Sergey Alexandrovich Kryukov 215
4 Peter Leow 180
0 Sergey Alexandrovich Kryukov 9,440
1 OriginalGriff 5,618
2 Peter Leow 4,280
3 Maciej Los 3,540
4 Abhinav S 3,363


Advertise | Privacy | Mobile
Web03 | 2.8.140415.2 | Last Updated 18 Nov 2011
Copyright © CodeProject, 1999-2014
All Rights Reserved. Terms of Use
Layout: fixed | fluid