Click here to Skip to main content
15,895,746 members
Articles / Web Development / ASP.NET

A Web Spider Library in C#

Rate me:
Please Sign up or sign in to vote.
4.28/5 (40 votes)
18 Sep 2007CPOL2 min read 265.3K   15.4K   170  
An article about a spider library to grab websites and store them locally
namespace Zeta.WebSpider.Spider
{
	#region Using directives.
	// ----------------------------------------------------------------------
	// ----------------------------------------------------------------------
	#endregion

	/////////////////////////////////////////////////////////////////////////

	/// <summary>
	/// Information about HTML elements that are links.
	/// </summary>
	internal sealed class LinkElement
	{
		#region Private methods.
		// ------------------------------------------------------------------

		/// <summary>
		/// Constructor.
		/// </summary>
		private LinkElement(
			UriType linkType,
			string name,
			params string[] attributes )
		{
			LinkType = linkType;
			Name = name;
			Attributes = attributes;
		}

		// ------------------------------------------------------------------
		#endregion

		#region Public properties.
		// ------------------------------------------------------------------

		/// <summary>
		/// This list was taken from the Perl module 'HTML-Tagset-3.03\blib\lib\HTML\Tagset.pm',
		/// '%linkElements' hash.
		/// </summary>
		public static readonly LinkElement[] LinkElements =
			new LinkElement[] 
		{ 
			new LinkElement( UriType.Content, @"a", "href" ),
			new LinkElement( UriType.Resource, @"applet", "archive", "codebase", "code" ),
			new LinkElement( UriType.Content, @"area", "href" ),
			new LinkElement( UriType.Resource, @"base", "href" ),
			new LinkElement( UriType.Resource, @"bgsound", "src" ),
			new LinkElement( UriType.Resource, @"blockquote", "cite" ),
			new LinkElement( UriType.Resource, @"body", "background" ),
			new LinkElement( UriType.Resource, @"del", "cite" ),
			new LinkElement( UriType.Resource, @"embed", "pluginspage", "src" ),
			new LinkElement( UriType.Resource, @"form", "action" ),
			new LinkElement( UriType.Content, @"frame", "src", "longdesc" ),
			new LinkElement( UriType.Content, @"iframe", "src", "longdesc" ),
			new LinkElement( UriType.Resource, @"ilayer", "background" ),
			new LinkElement( UriType.Resource, @"img", "src", "lowsrc", "longdesc", "usemap" ),
			new LinkElement( UriType.Resource, @"input", "src", "usemap" ),
			new LinkElement( UriType.Resource, @"ins", "cite" ),
			new LinkElement( UriType.Resource, @"isindex", "action" ),
			new LinkElement( UriType.Resource, @"head", "profile" ),
			new LinkElement( UriType.Resource, @"layer", "background", "src" ),
			new LinkElement( UriType.Content, @"link", "href" ),
			new LinkElement( UriType.Resource, @"object", "classid", "codebase", "data", "archive", "usemap" ),
			new LinkElement( UriType.Resource, @"q", "cite" ),
			new LinkElement( UriType.Resource, @"script", "src", "for" ),
			new LinkElement( UriType.Resource, @"table", "background" ),
			new LinkElement( UriType.Resource, @"td", "background" ),
			new LinkElement( UriType.Resource, @"th", "background" ),
			new LinkElement( UriType.Resource, @"tr", "background" ),
			new LinkElement( UriType.Resource, @"xmp", "href" ) 
		};

		/// <summary>
		/// The type of link.
		/// </summary>
		public readonly UriType LinkType;

		/// <summary>
		/// The name of the tag.
		/// </summary>
		public readonly string Name;

		/// <summary>
		/// The attributes that contain the link.
		/// </summary>
		public readonly string[] Attributes;

		// ------------------------------------------------------------------
		#endregion
	}

	/////////////////////////////////////////////////////////////////////////
}

By viewing downloads associated with this article you agree to the Terms of Service and the article's licence.

If a file you wish to view isn't highlighted, and is a text file (not binary), please let us know and we'll add colourisation support for it.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Chief Technology Officer Zeta Software GmbH
Germany Germany
Uwe does programming since 1989 with experiences in Assembler, C++, MFC and lots of web- and database stuff and now uses ASP.NET and C# extensively, too. He has also teached programming to students at the local university.

➡️ Give me a tip 🙂

In his free time, he does climbing, running and mountain biking. In 2012 he became a father of a cute boy and in 2014 of an awesome girl.

Some cool, free software from us:

Windows 10 Ereignisanzeige  
German Developer Community  
Free Test Management Software - Intuitive, competitive, Test Plans.  
Homepage erstellen - Intuitive, very easy to use.  
Offline-Homepage-Baukasten

Comments and Discussions