Simple Sitemaps in ASP.NET

• Download demo project - 6.7 KB

Introduction

The simple sitemap library is, as stated, an easy way to add sitemap support to your ASP.NET application. Just add the SitemapLib.cs file to your project and you are ready to go. The library implements the sitemaps.org standard for creating an XML sitemap file, and includes Ping support for all the search engines that currently support it: Google.com, Yahoo.com and Ask.com.

Please feel free to modify, reuse or redistribute the code as you see fit.

Background on Sitemaps

The sitemap standard was created by Google, Yahoo and Microsoft to provide an easy way for web site owners to inform search engines about pages on their site that are available for crawling.

Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support sitemaps to pick up all URLs in the sitemap and learn about those URLs using the associated metadata. Using sitemaps does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site.

Using SitemapLib

The library supports two main aspects of the standard, creating XML sitemaps, and Pinging the major search engines to notify them of updates in the sitemap.

Creating an XML Sitemap

The code snippet below demonstrates all the code required to create a sitemap using the library. Since the sitemap will be an XML file, it is better to use a "Generic Handler" file type (ashx) than an ASPX file. ASPX is optimized to generate HTML files, which doesn't work in this case. The ASHX file on the other hand, allows you to create any type of file you would like.

using System;
using stem.Web;

using SitemapLib;

public class Handler : IHttpHandler 
{
    public void ProcessRequest (HttpContext context)
    {
        SitemapLib.Sitemap sitemap = new SitemapLib.Sitemap();

        sitemap.AddLocation("http://mysite.com/default.aspx");
        sitemap.AddLocation(http://mysite.com/products/shoes/default.aspx, 
                                DateTime.Today);
        sitemap.AddLocation(http://mysite.com/products/coats/coat.aspx?id=3, 
                DateTime.Today, "0.8", ChangeFrequency.Monthly);

        context.Response.ContentType = "text/xml";
        context.Response.Write(sitemap.GenerateSitemapXML());
    }

    public bool IsReusable 
    {
        get 
        {
            return false;
        }
    }
}

Once you have instantiated a new SitemapLib.Sitemap class, you can use one of the many versions of the AddLocation() method to add URLs to your sitemap, depending on which metadata you would like to include in your sitemap file.

Note

The library validates the input URLs and the output file for common issues like malformed URLs, max URL length, max Filesize length, etc. To be a good programmer, you should use try{} catch(Exception e) {} and handle all errors thrown. Failure to manage these errors could mean that the search engines disregard your sitemap file.

Notifying Search Engines with a Ping

You have the option to notify each search engine that supports sitemaps of updates by using an HTTP Ping command. The following code shows you how to use the SitemapLib to ping all the Search Engines that support it.

// Replace with the fully qualified domain name of your sitemap file, 
// and your yahoo application id.
// You can provision one here: 
// http://developer.yahoo.com/search/siteexplorer/V1/updateNotification.html

Sitemap.Ping("http://mysite.com/sitemap.ashx", "Yahoo Application ID");

The first parameter should be the fully qualified path to your sitemap.xml file. Note that your XML file does not need to have an "XML" extension to be valid. The only odd thing about using the Ping is that Yahoo breaks the standard by requiring that developers include an application id with each ping. You can easily provision one here: http://developer.yahoo.com/search/siteexplorer/V1/updateNotification.html and include it in your program.

Notes on Implementing Sitemaps on your Website

• Don't forget to add the "sitemap:" directive to your Robots.txt file

  Instead of registering your sitemap with every search engine, you can support all of them by adding one line to your Robots.txt file, "Sitemap: <fully qualified path to your sitemap file, or sitemap index file>".

• Sitemaps are a suggestion, not a command

  The search engines will take all of the information provided by your sitemaps as one of many inputs into determining what pages should be included in their index, what is the relative priority of each page on your site, and how frequently your pages should be crawled.

• Generally, sitemaps will not impact your page rank

  They are most likely to affect the number of pages that are crawled and indexed, as well as the speed at which they are indexed. However, there are some cases where sitemaps have significantly impacted the page rank of a site, and that is when the sites were not getting fully crawled, and the sitemap led to the inclusion of some really good content into the index.

• Give your converting pages the highest priority

  Product pages, newsletter signup pages and other pages that generate the most value from your customers should receive the highest priority. Also include the pages that you know your customers will be the most interested in.

• Don't give all your pages 1.0 priority

  This just tells all the search engines that all your pages are of equal importance, it doesn't help boost the importance of any of your pages specifically.

• Give sitemaps time to see impact

  You may not see an impact right away from implementing sitemaps, but that doesn't mean that they are not being used. Search engines are still optimizing their implementations and figure out how exactly to use all of this information.