Microsoft Content Management Server Crawl Page for Search

• Download source files - 8.56 Kb

Introduction

This Crawl Results user control provides a start page for a search engine to crawl a Content Management Server (MCMS) web site.

For search purposes, it is recommended to hide a site's navigation menus when the browser user agent is detected to be a robot. This prevents words in the navigation menus from appearing in every page of the search result when a user searches for those words. With the navigation menus turned off, the search crawler will have no way to crawl through all the pages. This user control generates links to channels and postings, and allows a search engine to recursively crawl through all the pages.

As the user control does not generate links to all the channels and postings in one go, it is scalable to large sites. A meta robot tag of NOINDEX,FOLLOW should be included in the crawl page using the user control so that the crawler will not index the channel and posting lists but follow them.

Both C# and VB.NET versions are included.

Installation

1. Copy CrawlResults.ascx and its code-behind file to the user control directory in your MCMS site.
2. Create an ASPX page and insert CrawlResults.ascx. Change the StartChannelPath property to the root of your MCMS site. For example:

   <%@ Register TagPrefix="uc1" TagName="CrawlResults" 
               Src="~/UserControls/CrawlResults.ascx" %>
   . . .
   . . .
   <uc1:CrawlResults id="CrawlResults" 
     StartChannelPath="/Channels/WoodgroveNet" runat="server">
   </uc1:CrawlResults>

3. Add the following meta tag to the HTML header in the ASPX page:

   <meta name="ROBOTS" CONTENT="NOINDEX,FOLLOW">

4. Set the start page of the content source in your search engine to the URL of your ASPX page. For example, http://<server name>/<site name>/crawlpage.aspx.

History

• V1.0 - 2004.09.05 - Base.
• V1.1 - 2005.02.12 - Converted page to user control. Added option to specify the starting channel path.
• V1.2 - 2006.05.14 - ASP.NET 2.0 version added.