Click here to Skip to main content
Click here to Skip to main content

Simple Sitemaps in ASP.NET

, 21 Apr 2007
Rate this:
Please Sign up or sign in to vote.
Easiest way to create sitemaps for Google, Yahoo, and Ask. Provides a simple class which implements an XML sitemap file and the search engine ping.

Introduction

The simple sitemap library is, as stated, an easy way to add sitemap support to your ASP.NET application. Just add the SitemapLib.cs file to your project and you are ready to go. The library implements the sitemaps.org standard for creating an XML sitemap file, and includes Ping support for all the search engines that currently support it: Google.com, Yahoo.com and Ask.com.

Please feel free to modify, reuse or redistribute the code as you see fit.

Background on Sitemaps

The sitemap standard was created by Google, Yahoo and Microsoft to provide an easy way for web site owners to inform search engines about pages on their site that are available for crawling.

Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support sitemaps to pick up all URLs in the sitemap and learn about those URLs using the associated metadata. Using sitemaps does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site.

Using SitemapLib

The library supports two main aspects of the standard, creating XML sitemaps, and Pinging the major search engines to notify them of updates in the sitemap.

Creating an XML Sitemap

The code snippet below demonstrates all the code required to create a sitemap using the library. Since the sitemap will be an XML file, it is better to use a "Generic Handler" file type (ashx) than an ASPX file. ASPX is optimized to generate HTML files, which doesn't work in this case. The ASHX file on the other hand, allows you to create any type of file you would like.

using System;
using stem.Web;

using SitemapLib;

public class Handler : IHttpHandler 
{
    public void ProcessRequest (HttpContext context)
    {
        SitemapLib.Sitemap sitemap = new SitemapLib.Sitemap();

        sitemap.AddLocation("http://mysite.com/default.aspx");
        sitemap.AddLocation(http://mysite.com/products/shoes/default.aspx, 
                                DateTime.Today);
        sitemap.AddLocation(http://mysite.com/products/coats/coat.aspx?id=3, 
                DateTime.Today, "0.8", ChangeFrequency.Monthly);

        context.Response.ContentType = "text/xml";
        context.Response.Write(sitemap.GenerateSitemapXML());
    }

    public bool IsReusable 
    {
        get 
        {
            return false;
        }
    }
}

Once you have instantiated a new SitemapLib.Sitemap class, you can use one of the many versions of the AddLocation() method to add URLs to your sitemap, depending on which metadata you would like to include in your sitemap file.

Note

The library validates the input URLs and the output file for common issues like malformed URLs, max URL length, max Filesize length, etc. To be a good programmer, you should use try{} catch(Exception e) {} and handle all errors thrown. Failure to manage these errors could mean that the search engines disregard your sitemap file.

Notifying Search Engines with a Ping

You have the option to notify each search engine that supports sitemaps of updates by using an HTTP Ping command. The following code shows you how to use the SitemapLib to ping all the Search Engines that support it.

// Replace with the fully qualified domain name of your sitemap file, 
// and your yahoo application id.
// You can provision one here: 
// http://developer.yahoo.com/search/siteexplorer/V1/updateNotification.html

Sitemap.Ping("http://mysite.com/sitemap.ashx", "Yahoo Application ID");

The first parameter should be the fully qualified path to your sitemap.xml file. Note that your XML file does not need to have an "XML" extension to be valid. The only odd thing about using the Ping is that Yahoo breaks the standard by requiring that developers include an application id with each ping. You can easily provision one here: http://developer.yahoo.com/search/siteexplorer/V1/updateNotification.html and include it in your program.

Notes on Implementing Sitemaps on your Website

  • Don't forget to add the "sitemap:" directive to your Robots.txt file

    Instead of registering your sitemap with every search engine, you can support all of them by adding one line to your Robots.txt file, "Sitemap: <fully qualified path to your sitemap file, or sitemap index file>".

  • Sitemaps are a suggestion, not a command

    The search engines will take all of the information provided by your sitemaps as one of many inputs into determining what pages should be included in their index, what is the relative priority of each page on your site, and how frequently your pages should be crawled.

  • Generally, sitemaps will not impact your page rank

    They are most likely to affect the number of pages that are crawled and indexed, as well as the speed at which they are indexed. However, there are some cases where sitemaps have significantly impacted the page rank of a site, and that is when the sites were not getting fully crawled, and the sitemap led to the inclusion of some really good content into the index.

  • Give your converting pages the highest priority

    Product pages, newsletter signup pages and other pages that generate the most value from your customers should receive the highest priority. Also include the pages that you know your customers will be the most interested in.

  • Don't give all your pages 1.0 priority

    This just tells all the search engines that all your pages are of equal importance, it doesn't help boost the importance of any of your pages specifically.

  • Give sitemaps time to see impact

    You may not see an impact right away from implementing sitemaps, but that doesn't mean that they are not being used. Search engines are still optimizing their implementations and figure out how exactly to use all of this information.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

Nathan Buggia
Marketing
United States United States
Hello, I have been working in various aspects of web technology since my first web dev job in 1997, doing PERL and CGI. Since then I have worked in Java, C++/CGI (no, really), Systems Administration, Systems Architecture, and finally moved into Marketing for Microsoft's Live Search.
 
You can also find me at my blog: nathanbuggia.com

Comments and Discussions

 
GeneralVB code to use XmlTextWriter instead of StringBuilder: Pinmembery2kisdead19-Sep-07 8:13 
GeneralVB Translation PinmemberMike Joseph20-Jun-07 7:54 
Here's my VB.NET translation if it saves anyone any time -
 
Thanks for the Article!!
 
-------------------------------------------------------------------------
 
Imports Microsoft.VisualBasic
Imports System.Collections.Generic
Imports System.Net
 
' Original from: http://www.codeproject.com/aspnet/simplesitemaps.asp
Namespace SiteMapLib
 
      Public Enum ChangeFrequency
            Always = 0
            Hourly
            Daily
            Weekly
            Monthly
            Yearly
            Never
            DontUseThisField
      End Enum
 
#Region " SiteMapItem Class "
      Public Class SiteMapItem
            Private _loc As String
            Public Property Loc() As String
                  Get
                        Return _loc
                  End Get
                  Set(ByVal value As String)
                        _loc = value
                  End Set
            End Property
 
            Private _lastmod As DateTime
            Public Property LastMod() As DateTime
                  Get
                        Return _lastmod
                  End Get
                  Set(ByVal value As DateTime)
                        _lastmod = value
                  End Set
            End Property
 
            Private _priority As String
            Public Property Priority() As String
                  Get
                        Return _priority
                  End Get
                  Set(ByVal value As String)
                        _priority = value
                  End Set
            End Property
 
            Private _changeFreq As ChangeFrequency
            Public Property ChangeFreq() As ChangeFrequency
                  Get
                        Return _changeFreq
                  End Get
                  Set(ByVal value As ChangeFrequency)
                        _changeFreq = value
                  End Set
            End Property
 
            Public Sub New()
            End Sub
      End Class
 
#End Region
 
#Region " SiteMap Class "
      Public Class SiteMap
 
            Public SiteMapList As List(Of SiteMapItem) = Nothing
 
            Private Function _entityEscape(ByVal s As String) As String
                  Return s.Replace("&", "&amp;").Replace("'", "&apos;").Replace("""", "&quot;").Replace(">", "&gt;").Replace("<", "&lt;")
            End Function
 
            Public Sub AddLocation(ByVal location As String)
                  Me.AddLocation(location, New DateTime(0), "", ChangeFrequency.DontUseThisField)
            End Sub
 
            Public Sub AddLocation(ByVal location As String, ByVal lastmod As DateTime)
                  Me.AddLocation(location, lastmod, "", ChangeFrequency.DontUseThisField)
            End Sub
 
            Public Sub AddLocation(ByVal location As String, ByVal lastmod As DateTime, ByVal changeFreq As ChangeFrequency)
                  Me.AddLocation(location, lastmod, "", changeFreq)
            End Sub
 
            Public Sub AddLocation(ByVal location As String, ByVal lastmod As DateTime, ByVal priority As String, ByVal changeFreq As ChangeFrequency)
                  Dim item As New SiteMapItem()
                  item.Loc = location
                  item.LastMod = lastmod
                  item.Priority = priority
                  item.ChangeFreq = changeFreq
 
                  Me.AddLocation(item)
            End Sub
 
            Public Sub AddLocation(ByVal item As SiteMapItem)
                  If Me.SiteMapList Is Nothing Then SiteMapList = New List(Of SiteMapItem)
                  Me.SiteMapList.Add(item)
            End Sub
 
            Public Function GenerateSiteMapXML() As String
                  Dim sb As New StringBuilder("<?xml version=""1.0"" encoding=""UTF-8""?>")
                  sb.Append("<urlset xmlns=""http://www.sitemaps.org/schemas/sitemap/0.9"">")
 
                  ' ERROR CHECK: see if there are more than 50k URLs in file
                  If SiteMapList.Count > 50000 Then Throw New Exception("Sitemap file cannot contain more than 50,000 URLs. Refer to http://sitemaps.org for details")
 
                  For Each item As SiteMapItem In Me.SiteMapList
                        ' ERROR CHECK: Make sure a URL was entered
                        If item.Loc.Length < 9 Then
                              Throw New Exception("Sitemap entry must include URL. Refer to http://sitemaps.org for details")
                        End If
 
                        ' ERROR CHECK: URL must include protocol (http, https)
                        If (Not item.Loc.Substring(0, 7).ToLower().Equals("http://") And _
                              Not item.Loc.Substring(0, 8).ToLower().Equals("https://") And _
                              Not item.Loc.Substring(0, 7).ToLower().Equals("feed://")) Then
 
                              Throw New Exception("Sitemap URLs must include protocol (e.g. http://). Refer to http://sitemaps.org for details")
                        End If
 
                        ' ERROR CHECK: URL must be smaller than 2048 characters
                        If item.Loc.Length >= 2048 Then
                              Throw New Exception("Sitemap URLs cannot have more than 2048 characters. Refer to http://sitemaps.org for details")
                        End If
 
                        sb.Append("<url>")
 
                        ' LOCATION FIELD
                        sb.Append("<loc>")
                        sb.Append(_entityEscape(item.Loc))
                        sb.Append("</loc>")
 
                        ' LAST MODIFIED FIELD
                        If (0L <> item.LastMod.Ticks) Then
                              sb.Append("<lastmod>")
                              sb.Append(item.LastMod.ToString("yyyy-MM-dd"))
                              sb.Append("</lastmod>")
                        End If
 
                        ' CHANGE FREQUENCY FIELD
                        If (ChangeFrequency.DontUseThisField <> item.ChangeFreq) Then
                              sb.Append("<changefreq>")
 
                              Select Case item.ChangeFreq
                                    Case ChangeFrequency.Always
                                          sb.Append("always")
                                          Exit Select
                                    Case ChangeFrequency.Daily
                                          sb.Append("daily")
                                          Exit Select
                                    Case ChangeFrequency.Hourly
                                          sb.Append("hourly")
                                          Exit Select
                                    Case ChangeFrequency.Monthly
                                          sb.Append("monthly")
                                          Exit Select
                                    Case ChangeFrequency.Never
                                          sb.Append("never")
                                          Exit Select
                                    Case ChangeFrequency.Weekly
                                          sb.Append("weekly")
                                          Exit Select
                                    Case ChangeFrequency.Yearly
                                          sb.Append("yearly")
                                          Exit Select
                              End Select
 
                              sb.Append("</changefreq>")
                        End If
 
                        If item.Priority.Length > 0 Then
                              sb.Append("<priority>")
                              sb.Append(item.Priority)
                              sb.Append("</priority>")
                        End If
 
                        sb.Append("</url>")
                  Next
 
                  sb.Append("</urlset>")
 
                  If sb.Length > 10485760 Then
                        Throw New Exception("Sitemap file cannot be larger than 10MB. Refer to http://sitemaps.org for details")
                  End If
 
                  Return sb.ToString
            End Function
 
            Public Sub Ping(ByVal sitemapFileURL As String, ByVal yahooAppID As String)
                  Dim STD_PING_PATH As String = "/ping?sitemap=" + sitemapFileURL
 
                  Try
                        Dim request As WebRequest = System.Net.HttpWebRequest.Create("http://www.google.com/webmasters/tools" + STD_PING_PATH)
                        Dim response As WebResponse = request.GetResponse()
                  Catch e As Exception
                        ' TODO: handle this error!
                        Throw e
                  End Try
 
                  ' Yahoo:
                  Try
                        Dim request As WebRequest = System.Net.HttpWebRequest.Create("http://search.yahooapis.com/SiteExplorerService/V1/updateNotification?appid=" + yahooAppID + "&url=" + sitemapFileURL)
                        Dim response As WebResponse = request.GetResponse()
                  Catch e As Exception
                        ' TODO: handle this error!
                        Throw e
                  End Try
 
                  ' ASK.COM:
                  Try
                        Dim request As WebRequest = System.Net.HttpWebRequest.Create("http://submissions.ask.com" + STD_PING_PATH)
                        Dim response As WebResponse = request.GetResponse()
                  Catch e As Exception
                        ' TODO: handle this error!
                        Throw e
                  End Try
            End Sub
      End Class
 
#End Region
 
End Namespace

 
Mike Joseph
GeneralRe: VB Translation PinmemberNathan Buggia20-Jun-07 8:06 
GeneralRe: VB Translation PinmemberWhittet25-Jun-08 9:22 
Question'&' not allowed Pinmemberjariaman27-May-07 20:32 
AnswerRe: '&' not allowed PinmemberNathan Buggia27-May-07 21:54 
GeneralSuggestions PinmemberRichard Deeming24-Apr-07 7:32 
GeneralRe: Suggestions PinmemberNathan Buggia24-Apr-07 8:04 
GeneralRe: Suggestions [modified] PinmemberRichard Deeming25-Apr-07 1:21 
NewsSource Code Posted! PinmemberNathan Buggia24-Apr-07 5:57 
QuestionRe: Source Code Posted! PinmemberSteveC-A94-May-07 5:20 
QuestionWhere's the beef? PinmemberPCBender24-Apr-07 5:14 
QuestionWhere can one access the SitemapLib.cs file? PinmemberTheBigBadWolf23-Apr-07 21:04 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web04 | 2.8.140709.1 | Last Updated 22 Apr 2007
Article Copyright 2007 by Nathan Buggia
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid