Click here to Skip to main content
13,145,774 members (48,217 online)
Click here to Skip to main content
Add your own
alternative version


33 bookmarked
Posted 26 Dec 2009

A simple crawler class

, 27 Dec 2009
Rate this:
Please Sign up or sign in to vote.
A multi-threaded crawler that downloads pages and saves them localy while preserving the site tree structure.


"MyCrawler" is a class that is written in C# and functions as a "Crawler", a program that crawls a website given a root URL (or several roots) and downloads them and any links within those pages, using multiple threads, filters, and depth for precise and fast downloading.


This class was written as part of a larger project for data mining.


Besides the main class "CrawlerGeneric", there are a few more helper classes:

  • Network - inherits the "WebClient" class and overloads the "downloadstring" method for more specific tasks.
  • IO - responsible for saving pages locally.
  • MyRegex - helper class that includes various methods to manipulate strings.
  • Compresser - is used by the network class for handling compressed data.

Using the code

To use the crawler, we need to first create an instance of "CrawlerGeneric"; however, it is recommended to use it as a base class and create your own class for implementing the "ExtractLinks" and "Filter" methods (described below):

public class MyCrawler : CrawlerGeneric
    protected override List<string> ExtractLinks(string pageData,string url)
        List<string> links = BuiltInExtractLinks(pageData,url);
        //Do some sort of logic here
        return links;

    protected override bool Filter(string url)
        url = url.ToLower();
        if (url.Contains("search?") && url.Contains("&start="))
            return false;
        return true;

The "ExtractLinks" method allows you to get the downloaded page data and its URL and add or remove links from the links list. The links list holds the list of URLs that were extracted from the page. BuiltInExtractLinks is a private function that extracts all the URLs inside the "href" attribute in the page.

The "Filter" method examines each URL in the link list; only URLs that return false will be downloaded.

Create an instance of MyCrawler

After writing our own class, we instantiate it and set its properties.

MyCrawler crawler = new MyCrawler();
crawler.CrawlerDepth = 2;
crawler.NumOfConcurrentThreads = 10;
crawler.OutputDirectory = "c:\\test";
crawler.Roots = new string[] { @"" & _ 
   "hl=en&lr=&rlz=1G1GGLQ_IWIL297&q=codeproject&start=0&sa=N" };
crawler.AllowCompression = false;

//Time for thread to sleep when there is no url's to handle
crawler.ThreadIdleTime = 500;
//Time for thread to sleep after downloading a page
crawler.ThreadSleepTime = 500; 

The "AllowComperssion" property allows us to choose if the data that is downloaded could be downloaded using the "gzip/deflate" compressors. In this case, I set it to false because I am downloading pages from Google, and I noticed that compressed data that is sent from Google is compressed using the zip method and not gzip, thus receiving an exception "The magic number in GZip header is not correct. Make sure you are passing in a GZip stream ".

Hooking up events

The class can be consumed by any UI app. So, it uses an event to alert changes. Also, it has an event to alert when the download process has ended.


crawler.DataCanged += new 

crawler.Finished += new EventHandler<Analayza.Crawler.

//Check connection to internet
if (!crawler.IsConnectedToInternet())
    Console.WriteLine("Not connected to the internet");

//Start crawling

//dont exit until crawler has finished
while (fin == false)

static void crawler_Finished(object sender, 
  Analayza.Crawler.Events.FinishedDownloadingEventArgs e)
    fin = true;

static void crawler_DataCanged(object sender, 
  Analayza.Crawler.Events.MyEventArgs e)

Points of interest

Because the crawler downloads the site tree and uses the URL path as the file name, there might be cases where the entire path on the local disk exceeds 260 characters, which is the Windows file system limit for a path length. So the crawler uses a method called MyRegex.ToValidWindowsPath to transform the URL to a valid Windows path (excluding illegal characters) and if needed, cutting the name (limiting to 250, just in case..).

public static string ToValidWindowsPath(string directory, string file)
    string newfile = ToValidWindowsFileName(file);
    string fullPath = directory + newfile;

    if (fullPath.Length > 250)
        if (directory.Length > 230)
            return "";

        int fileNewLength = 248 - directory.Length; //minimum 18 characters

        if (newfile.Length > 10)
            newfile = newfile.Substring(0, 10);

        Random rand = new Random();
        fullPath = directory + newfile + rand.Next(0, 400000).ToString();
        return fullPath;
    return fullPath;

Thank to..

Thanks to gizmo ... :)


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Haim Nachum
Software Developer
Israel Israel
No Biography provided

You may also be interested in...

Comments and Discussions

GeneralMy vote of 3 Pin
Mohammad Dayyan28-Dec-09 7:30
memberMohammad Dayyan28-Dec-09 7:30 
GeneralMissing code Pin
Steve Hansen27-Dec-09 0:08
memberSteve Hansen27-Dec-09 0:08 
GeneralRe: Missing code Pin
Haimbert27-Dec-09 0:35
memberHaimbert27-Dec-09 0:35 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.170915.1 | Last Updated 27 Dec 2009
Article Copyright 2009 by Haim Nachum
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid