Click here to Skip to main content
12,954,522 members (53,359 online)
Click here to Skip to main content
Add your own
alternative version


50 bookmarked
Posted 25 Nov 2005

Google Parser

, 25 Nov 2005
Rate this:
Please Sign up or sign in to vote.
Parsing Google pages using RegularExpressions in C#.

PGoogle in action, searching for Google Parsers in CodeProject


Google is definitely one of the most useful websites on the net. I am using it every day and like other things I use frequently, I intended to customize it to my personal changing needs. As a developer the first thing that came to my mind to automate my activities was that I needed a class representing this search engine. When I looked into their web site I saw that they provide a very nice API which is accessible through Web Services.

For some strange reason, I couldn't access the web service out of the company from behind the firewall... So I had to look into other options. Looking into CodeProject's web site, I figured out that I am not the only one interested in this subject. There has been already some nice articles written about this matter. There is a nice code provided by Stuart Konen in C++ but I was looking for a .NET assembly. Also Kisilevich Slava is providing code with a much better interface and even a more powerful engine which gets its results from different search engines. But (s)he uses another HTML parser which makes it dependent on it and I wanted to have full control of the parsing of the page.

So, I came up with this parser...


Providing a Google interface has two challenges involved. First, you should write a code to establish a connection to send your HTTP request and get your response back. This is very easy when you work at home where the permissions are granted automatically. At the office there is a proxy which blocks the unattended internet access. So the issue here is to provide the DefaultCredentials as Internet Explorer would do. Fortunately Mr. Gates has provided an easy solution for this:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Proxy = WebProxy.GetDefaultProxy();
request.Proxy.Credentials = System.Net.CredentialCache.DefaultCredentials;

Once this is done, you can request anything IExplorer can. For more information on DefaultCredentials see the MSDN.

The second challenge was to parse the response. This is the tricky part. Some do this by converting the HTML to XML and then use the XML parsers. Others use COM and use HtmlDocument to access the tags as DOM objects. Some others like me prefer parsing it using RegularExpressions. The last approach is surely the least readable code but surely fast and elegant.

Describing Google's Search Object


Each result item has the following attributes:

  • link to the website
  • description of the found item
  • some fragment of text having one or more words from the query
  • a link to Cache
  • and a link to Similar pages

from which the last two parts are not supported in my demo.

This brings us to our first class Item which gets constructed as follows:

public Item(string link, string description, string text)
    this._Link = link;
    this._Description = description;
    this._Text = text;


The Searcher class provides the following properties:

  • Count
  • From
  • To
  • ItemsPerPage
  • Url and
  • Results

Of properties, only the ItemsPerPage is read/write, the rest are read-only.

It has three public methods which all help to initiate the search. The first one which should also be called first is the Search method which is to be given a query parameter.

public void Search(string query)

Once you call this method, the class will get the response and parse the page to get Count and Results. The you can call subsequent search methods exactly like you would do on the Google page.

  • public void Search(long from)
  • public void SearchNext()


As I was working with this project, I realized that Google developers are changing the HTML layout more often than they change the logo on the front page. So I had to make sure my code was still working. But then they never promised anyone to keep the same HTML layout. Which makes this project very unstable. In other words nothing guarantees that it will work after a while. That is why I decided to have a project with full control over the whole code and not using any third party library. That way, I can modify the code easily to accommodate any changes in the Google HTML format. The whole parsing process has been split into three sections. First I have to get the counts and I m doing it in the GetCounts method. Then I get the division of results and parse the items out of it in a loop implemented in GetResults. For each item, I parse the HTML to get its properties and that happens within ParseItem.

Using the Code

In my demo I have a form to query Google and show the results in a list box. I have also provided a WebBrowser control to see how it looks within IExplorer. Also there is a link on the title to initiate IExplorer outside of the application. The code using this class is pretty simple.

private Searcher google = new Searcher();
    google.ItemsPerPage = nudItemsPerPage.Value ;
    google.Search( txtSearch.Text);
    if (google.Results==null) return;

    btnNext.Enabled= true;
catch (Exception ex)
lstLinks.DataSource = google.Results ;
lstLinks.DisplayMember = "Description";

Last Words

There are plenty of ways to parse your HTML code. Depending on the problem you will need to choose one. Working with regular expressions is the most fascinating one and there is undoubtedly a lot to explore in that area.


  • I initiated the first version of this code on 13-Nov-2005.
  • Submitted the first version to CodeProject on 25-Nov-2005.


This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


About the Author

Asghar Panahy
Systems Engineer
Netherlands Netherlands
I am a Microsoft certified application developer and work on Microsoft platform since 1993. I like to learn how Microsoft Systems have been built up and dig myself into different libraries in Win32 and .NET. I have worked with C++, VB6, Java, C#, ASP, ASP.Net and PHP.

You may also be interested in...

Comments and Discussions

GeneralNot working Pin
Divyan Kavdia21-Oct-08 17:57
memberDivyan Kavdia21-Oct-08 17:57 
GeneralThe construction of the project Pin
Asghar Panahy21-Oct-08 21:19
memberAsghar Panahy21-Oct-08 21:19 
GeneralRe: Not working Pin
Ali Kashanchi28-Sep-16 9:45
memberAli Kashanchi28-Sep-16 9:45 
Generalmay be format of Google results out of date Pin
tuanpm11-Jun-08 18:17
membertuanpm11-Jun-08 18:17 
GeneralRe: may be format of Google results out of date Pin
jsamper24-Sep-08 14:59
memberjsamper24-Sep-08 14:59 
GeneralRe: may be format of Google results out of date Pin
Asghar Panahy25-Sep-08 21:34
memberAsghar Panahy25-Sep-08 21:34 
Questionabout other search engines? Pin
~Samaneh~19-Mar-08 0:39
member~Samaneh~19-Mar-08 0:39 
AnswerRe: about other search engines? Pin
Asghar Panahy29-Mar-08 12:24
memberAsghar Panahy29-Mar-08 12:24 
QuestionIndex was out of range index Pin
Ehsan1201-Nov-07 16:32
memberEhsan1201-Nov-07 16:32 
NewsTest Google Parser Online Tool Pin
jp73129-Oct-07 16:51
memberjp73129-Oct-07 16:51 
GeneralCool , but ... Pin
DR_Mosan10-Jul-06 0:05
memberDR_Mosan10-Jul-06 0:05 
AnswerRe: Cool , but ... Pin
Asghar Panahy11-Jul-06 12:51
memberAsghar Panahy11-Jul-06 12:51 
GeneralVery Nice! Pin
redfish349-Dec-05 5:13
memberredfish349-Dec-05 5:13 
AnswerShell Document Viewer Pin
Asghar.Panahy10-Dec-05 14:31
memberAsghar.Panahy10-Dec-05 14:31 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.170525.1 | Last Updated 25 Nov 2005
Article Copyright 2005 by Asghar Panahy
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid