Click here to Skip to main content
Click here to Skip to main content
Go to top

Google Parser

, 25 Nov 2005
Rate this:
Please Sign up or sign in to vote.
Parsing Google pages using RegularExpressions in C#.

PGoogle in action, searching for Google Parsers in CodeProject

Introduction

Google is definitely one of the most useful websites on the net. I am using it every day and like other things I use frequently, I intended to customize it to my personal changing needs. As a developer the first thing that came to my mind to automate my activities was that I needed a class representing this search engine. When I looked into their web site I saw that they provide a very nice API which is accessible through Web Services.

For some strange reason, I couldn't access the web service out of the company from behind the firewall... So I had to look into other options. Looking into CodeProject's web site, I figured out that I am not the only one interested in this subject. There has been already some nice articles written about this matter. There is a nice code provided by Stuart Konen in C++ but I was looking for a .NET assembly. Also Kisilevich Slava is providing code with a much better interface and even a more powerful engine which gets its results from different search engines. But (s)he uses another HTML parser which makes it dependent on it and I wanted to have full control of the parsing of the page.

So, I came up with this parser...

Challenges

Providing a Google interface has two challenges involved. First, you should write a code to establish a connection to send your HTTP request and get your response back. This is very easy when you work at home where the permissions are granted automatically. At the office there is a proxy which blocks the unattended internet access. So the issue here is to provide the DefaultCredentials as Internet Explorer would do. Fortunately Mr. Gates has provided an easy solution for this:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Proxy = WebProxy.GetDefaultProxy();
request.Proxy.Credentials = System.Net.CredentialCache.DefaultCredentials;

Once this is done, you can request anything IExplorer can. For more information on DefaultCredentials see the MSDN.

The second challenge was to parse the response. This is the tricky part. Some do this by converting the HTML to XML and then use the XML parsers. Others use COM and use HtmlDocument to access the tags as DOM objects. Some others like me prefer parsing it using RegularExpressions. The last approach is surely the least readable code but surely fast and elegant.

Describing Google's Search Object

Item

Each result item has the following attributes:

  • link to the website
  • description of the found item
  • some fragment of text having one or more words from the query
  • a link to Cache
  • and a link to Similar pages

from which the last two parts are not supported in my demo.

This brings us to our first class Item which gets constructed as follows:

public Item(string link, string description, string text)
{
    this._Link = link;
    this._Description = description;
    this._Text = text;
}

Searcher

The Searcher class provides the following properties:

  • Count
  • From
  • To
  • ItemsPerPage
  • Url and
  • Results

Of properties, only the ItemsPerPage is read/write, the rest are read-only.

It has three public methods which all help to initiate the search. The first one which should also be called first is the Search method which is to be given a query parameter.

public void Search(string query)

Once you call this method, the class will get the response and parse the page to get Count and Results. The you can call subsequent search methods exactly like you would do on the Google page.

  • public void Search(long from)
  • public void SearchNext()

Parsing

As I was working with this project, I realized that Google developers are changing the HTML layout more often than they change the logo on the front page. So I had to make sure my code was still working. But then they never promised anyone to keep the same HTML layout. Which makes this project very unstable. In other words nothing guarantees that it will work after a while. That is why I decided to have a project with full control over the whole code and not using any third party library. That way, I can modify the code easily to accommodate any changes in the Google HTML format. The whole parsing process has been split into three sections. First I have to get the counts and I m doing it in the GetCounts method. Then I get the division of results and parse the items out of it in a loop implemented in GetResults. For each item, I parse the HTML to get its properties and that happens within ParseItem.

Using the Code

In my demo I have a form to query Google and show the results in a list box. I have also provided a WebBrowser control to see how it looks within IExplorer. Also there is a link on the title to initiate IExplorer outside of the application. The code using this class is pretty simple.

private Searcher google = new Searcher();
...
try
{
    google.ItemsPerPage = nudItemsPerPage.Value ;
    google.Search( txtSearch.Text);
    if (google.Results==null) return;

    btnNext.Enabled= true;
}
catch (Exception ex)
{
    MessageBox.Show(ex.Message);
}
lstLinks.DataSource = google.Results ;
lstLinks.DisplayMember = "Description";

Last Words

There are plenty of ways to parse your HTML code. Depending on the problem you will need to choose one. Working with regular expressions is the most fascinating one and there is undoubtedly a lot to explore in that area.

History

  • I initiated the first version of this code on 13-Nov-2005.
  • Submitted the first version to CodeProject on 25-Nov-2005.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Asghar Panahy
Systems Engineer
Netherlands Netherlands
I am a Microsoft certified application developer and work on Microsoft platform since 1993. I like to learn how Microsoft Systems have been built up and dig myself into different libraries in Win32 and .NET. I have worked with C++, VB6, Java, C#, ASP, ASP.Net and PHP.

Comments and Discussions

 
GeneralNot working PinmemberDivyan Kavdia21-Oct-08 17:57 
GeneralThe construction of the project PinmemberAsghar Panahy21-Oct-08 21:19 
Generalmay be format of Google results out of date Pinmembertuanpm11-Jun-08 18:17 
GeneralRe: may be format of Google results out of date Pinmemberjsamper24-Sep-08 14:59 
GeneralRe: may be format of Google results out of date PinmemberAsghar Panahy25-Sep-08 21:34 
Questionabout other search engines? Pinmember~Samaneh~19-Mar-08 0:39 
AnswerRe: about other search engines? PinmemberAsghar Panahy29-Mar-08 12:24 
QuestionIndex was out of range index PinmemberEhsan1201-Nov-07 16:32 
NewsTest Google Parser Online Tool Pinmemberjp73129-Oct-07 16:51 
GeneralCool , but ... PinmemberDR_Mosan10-Jul-06 0:05 
AnswerRe: Cool , but ... PinmemberAsghar Panahy11-Jul-06 12:51 
GeneralVery Nice! Pinmemberredfish349-Dec-05 5:13 
AnswerShell Document Viewer PinmemberAsghar.Panahy10-Dec-05 14:31 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140922.1 | Last Updated 25 Nov 2005
Article Copyright 2005 by Asghar Panahy
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid