Click here to Skip to main content
13,087,784 members (77,876 online)
Click here to Skip to main content
Add your own
alternative version


29 bookmarked
Posted 7 Jul 2009

Google Spider .NET

, 9 Jul 2009
Rate this:
Please Sign up or sign in to vote.
Search your keyword against your competitors


Regular expression engine.


This code is able to search for keywords on and return position result. It also supports multi keywords search and competitors list for comparing with your domain.

The project builds in three different parts:

  • core: This is the logic file containing support functions for the incoming date from the gui layer. It's also able to remove the dependency to specific gui, so you can customize it for any application type (web application, WPF).
  • keys match: The gui layer, all the checks and validations are made there.
  • keysmatch_setup: The setup project for the application.


What kind of experience do you get in this article:

  1. Use Webclient object
  2. Use Regular Expression mechanism
  3. Save files and settings
  4. Log viewer
  5. Setup project

Transfer Objects

The objects transfer between the layers and keep the parameters language in the project:

public class client_obj
string domain;
public string Domain { get { return domain; } set { domain = value; } }
ArrayList keyword = new ArrayList();
public ArrayList KeyWord { get { return keyword; } set { keyword = value; }}
ArrayList competiters = new ArrayList();
public ArrayList Competiters { get { return competiters; } set { competiters = value; } }
public class keyword_obj
string keyword;
public string KeyWord { get { return keyword; } set { keyword = value; } }
int position;
public int Position { get { return position; } set { position = value; } }
int page;
public int Page { get { return page; } set { page = value; } }
public class competitors
string domain;
public string Domain { get { return domain; } set { domain = value; } }

ArrayList keyword = new ArrayList();
public ArrayList KeyWord { get { return keyword; } set { keyword = value; } }
  • competiters: Represents the competitors values
  • keyword_obj: Represents the keywords values
  • client_obj: Contains the client search values

Using the HtmlParse Function

The program is based on simple Regular Expression function, the function creating arraylist for the return results of your search by getting the HTML page from the WebClient object.

private ArrayList HtmlParse(string html)
Match m;
ArrayList LinkArray = new ArrayList();
string HRefPattern = "<h3 class=r><a href\\s*=\\s*(?:\"(?<1>[^\"]*)\"|(?<1>\\S+))";
m = Regex.Match(html, HRefPattern, RegexOptions.IgnoreCase);
while (m.Success)
string TempLink = m.Groups[1].Value;
m = m.NextMatch();
return LinkArray;

HrefPattern var contains regular expression language (what to search in string; as you see I'm looking for domain address in H3 tag class. In Google, it represents the result address on the page.

Web Client Function

This function creates the connection for the site and gets a string containing all the generated HTML from the request site.

private string RuunerGoogleHtml(string key, string RunStep)
WebClient Client = new WebClient();
Client.Encoding = Encoding.GetEncoding("windows-1255");
string fixUrl = "" + 
	key + "&hl=iw&as_qdr=all&num=10&start=" + RunStep + "&sa=N";
string html = Client.Encoding.GetString(Client.DownloadData(fixUrl));
return html;

fixurl contains the search parameters in Google. You can see it in the address box in your explorer when you do a search.

Log Engine

The application creates log files for any search. You can save the output file for history and also save your settings search.

For more information on the functions, see class file_manager.cs.

Gui Layer

This gets the users' inputs from the controls and makes the request by sending the request parameters to the core layer:

private void btn_start_Click(object sender, EventArgs e)
if (chk_google.Checked)
if (txb_client_domain.Text != "http://")
if (txb_keyword.Text != "")
lbl_Status.Text = "";
myClient = null;
MyKeyWords = null;
myClient = new client_obj();
MyKeyWords = new ArrayList();
StopStatus = false;
googleWorker = new BackgroundWorker();
googleWorker.WorkerSupportsCancellation = true;
googleWorker.DoWork += new DoWorkEventHandler(googleWorker_DoWork);
googleWorker.RunWorkerCompleted += new RunWorkerCompletedEventHandler
lbl_Status.Text = "Enter keywords for search";
lbl_Status.Text = "Insert domain for search";
lbl_Status.Text = "Active your search engine";
void googleWorker_DoWork(object sender, DoWorkEventArgs e)
for (int a = 0; a < MyKeyWords.Count; a++)
if (!StopStatus)
dal_regex myRegex = new dal_regex();
keyword_obj mykey = new keyword_obj();
mykey = (keyword_obj)MyKeyWords[a];
for (int b = 0; b < TotalPage2Search; b++)
MethodInvoker Invoker = delegate
//lbl_Status.Text = "Search: " + tempKey.KeyWord + 
	" Page Position: " + b.ToString() + " Hit: " + hit + "- your domain result";
myClient = myRegex.CatchValues(myClient, mykey.KeyWord, b);
google_progress.Value = google_progress.Value + 1;
google_progress.Value = google_progress.Maximum;
lbl_page.Text = (b + 1).ToString();
lbl_Status.Text = "In progress...";
lbl_key.Text = mykey.KeyWord;

googleworker is a type of BackGroundWorker belonging to the Threads family. It starts a new instance of the search procedure according to the page search value to not affect the main thread of the program.


I hope to see more versions of this project.

Enjoy coding.

Don't forget to comment. :)


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Israel Israel
No Biography provided

You may also be interested in...

Comments and Discussions

GeneralCant seem to get any results Pin
rippo7-Jul-09 21:24
memberrippo7-Jul-09 21:24 
GeneralRe: Cant seem to get any results Pin
rippo7-Jul-09 21:26
memberrippo7-Jul-09 21:26 
GeneralRe: Cant seem to get any results Pin
Proxytype8-Jul-09 0:11
memberProxytype8-Jul-09 0:11 
GeneralRe: Cant seem to get any results Pin
gh0zt10-Jul-09 3:00
membergh0zt10-Jul-09 3:00 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.170813.1 | Last Updated 9 Jul 2009
Article Copyright 2009 by Proxytype
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid