Click here to Skip to main content
Click here to Skip to main content

A Web Spider Library in C#

, 18 Sep 2007 CPOL
Rate this:
Please Sign up or sign in to vote.
An article about a spider library to grab websites and store them locally

Sample Image - ZetaWebSpider.png

Don't fear, it's just a web spider Wink | ;-)

Introduction

Today, while looking through some older code, I came across a set of classes I wrote at the beginning of this year for a customer project.

The classes implement a basic web spider (also called "web robot" or "web crawler") to grab web pages (including resources like images and CSS), download them locally and adjust any resource hyperlinks to point to the locally downloaded resources.

While this article is not a full-featured article with detailed explanations as I usually like to write, I still want to put the code online with this short article. Maybe some reader can still take some ideas from this code and use it as a starting point for his own project.

Overview

The classes allow for synchronous as well as asynchronous download of the web pages, allowing multiple options to be specified like hyperlink-depth to follow and proxy settings.

The downloaded resources get their own new file names, based on the hash code of the original URL. I did this for simplifications (for me as the programmer).

To parse a document, I am using the SGMLReader DLL from the GotDotNet website.

Also, since I didn't need it for the project I wrote, the library does not care about "robots.txt" or throttling or other features.

Using the Code

The download for this article contains the library ("WebSpider") and a testing console application ("WebSpiderTest"). The testing application is rather short and should be rather easy to understand.

Basically, you do create an instance of the WebSiteDownloaderOptions class, configure several parameters, create an instance of the WebSiteDownloader class, optionally connect event handlers and then tell the instance to either start synchronously or asynchronously processing the given URL.

History

  • 2007-09-17: Fixed several issues
  • 2006-09-10: Initial release of the article

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Uwe Keim
Chief Technology Officer Zeta Producer Desktop CMS
Germany Germany
Uwe does programming since 1989 with experiences in Assembler, C++, MFC and lots of web- and database stuff and now uses ASP.NET and C# extensively, too. He has also teached programming to students at the local university.
 
In his free time, he does climbing, running and mountain biking. Recently he became a father of a cute boy.
 
Some cool, free software from us:
 
Free Test Management Software - Intuitive, competitive, Test Plans. Download now!  
Homepage erstellen - Intuitive, very easy to use. Download now!  
Send large Files online for free by Email
Offline-Homepage-Baukasten

Comments and Discussions

 
QuestionYou can also find another workaround here PinmemberWinston Gubantes5-Mar-14 5:56 
Generaldoesn't work correctly in indexof: based sites PinmemberArpit sharma2-Apr-11 5:27 
GeneralProblem In SGML PinmemberArpit sharma27-Dec-10 5:23 
GeneralIssue following links PinmemberStubbsPKS26-Oct-10 7:07 
AnswerRe: Issue following links PinmemberStubbsPKS26-Oct-10 8:31 
GeneralRe: Issue following links PinmvpUwe Keim26-Oct-10 9:41 
GeneralMy vote of 2 Pinmemberdaveauld20-Jun-10 0:51 
GeneralRe: My vote of 2 PinmemberKevin Yochum9-Aug-10 8:59 
GeneralRe: My vote of 2 PinmvpUwe Keim9-Aug-10 20:39 
Thank you very much for your kind words, Kevin!
My personal 24/7 webcam  
Free Test Management Software - Intuitive, competitive, Test Plans. Download now!  
Free Website Builder - Intuitive, very easy to use. Download now!

General3k Pinmembersongmei.lv@163.com7-Dec-09 1:03 
General.STATE File PinmemberMember 441033831-Oct-09 12:46 
GeneralMy vote of 1 Pinmemberbabakzawari20-Oct-09 1:23 
GeneralWeb Spider Issue PinmemberMember 4747242-Sep-09 23:17 
GeneralRe: Web Spider Issue PinsitebuilderUwe Keim2-Sep-09 23:29 
Generalxml site map PinmemberRohit_kakria10-Aug-09 2:51 
GeneralJust Links [modified] PinmemberSosyopat30-Jul-09 11:18 
GeneralRe: Just Links PinsitebuilderUwe Keim30-Jul-09 20:22 
GeneralThe remote server returned an error: (403) Forbidden Pinmemberkakakkakka21-Jul-09 2:00 
GeneralRe: The remote server returned an error: (403) Forbidden PinsitebuilderUwe Keim21-Jul-09 2:20 
Generalquestion plz Pinmembernaroqueen22-May-09 20:22 
Generalnice spider and nice code Pinmemberpsyhf21-Mar-09 21:03 
QuestionI am getting an unhandled exception while running the test project PinmemberMember 47118243-Jun-08 10:47 
GeneralException Running Test Pinmembercornix430-Apr-08 10:49 
Generalso good Pinmemberokzhuce28-Apr-08 21:18 
GeneralRe: so good PinsitebuilderUwe Keim28-Apr-08 21:43 
Generalhello PinmemberMember 451146418-Feb-08 15:48 
QuestionVery nice Pinmemberaaron_myers18-Sep-07 5:59 
AnswerRe: Very nice PinsitebuilderUwe Keim18-Sep-07 21:37 
GeneralRe: Very nice Pinmemberaaron_myers22-Oct-07 4:53 
QuestionRe: Very nice Pinmemberbabaa24-Feb-08 17:57 
GeneralGreat App Pinmemberrdissell1-Aug-07 6:14 
GeneralGreat job. But i met a problem PinmemberBlueLoveCyn14-Sep-06 22:28 
GeneralYour Project No Worky Pinmembereasy_coder12-Sep-06 4:04 
GeneralRe: Your Project No Worky PinsitebuilderUwe Keim12-Sep-06 4:19 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.1411019.1 | Last Updated 19 Sep 2007
Article Copyright 2006 by Uwe Keim
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid