Click here to Skip to main content
Click here to Skip to main content

Microsoft Content Management Server Crawl Page for Search

, 17 May 2006 CPOL
Rate this:
Please Sign up or sign in to vote.
Provides a start page for search engines to crawl a Content Management Server (MCMS) web site.

Sample Image - sample.gif

Introduction

This Crawl Results user control provides a start page for a search engine to crawl a Content Management Server (MCMS) web site.

For search purposes, it is recommended to hide a site's navigation menus when the browser user agent is detected to be a robot. This prevents words in the navigation menus from appearing in every page of the search result when a user searches for those words. With the navigation menus turned off, the search crawler will have no way to crawl through all the pages. This user control generates links to channels and postings, and allows a search engine to recursively crawl through all the pages.

As the user control does not generate links to all the channels and postings in one go, it is scalable to large sites. A meta robot tag of NOINDEX,FOLLOW should be included in the crawl page using the user control so that the crawler will not index the channel and posting lists but follow them.

Both C# and VB.NET versions are included.

Installation

  1. Copy CrawlResults.ascx and its code-behind file to the user control directory in your MCMS site.
  2. Create an ASPX page and insert CrawlResults.ascx. Change the StartChannelPath property to the root of your MCMS site. For example:
    <%@ Register TagPrefix="uc1" TagName="CrawlResults" 
                Src="~/UserControls/CrawlResults.ascx" %>
    . . .
    . . .
    <uc1:CrawlResults id="CrawlResults" 
      StartChannelPath="/Channels/WoodgroveNet" runat="server">
    </uc1:CrawlResults>
  3. Add the following meta tag to the HTML header in the ASPX page:
    <meta name="ROBOTS" CONTENT="NOINDEX,FOLLOW">
  4. Set the start page of the content source in your search engine to the URL of your ASPX page. For example, http://<server name>/<site name>/crawlpage.aspx.

History

  • V1.0 - 2004.09.05 - Base.
  • V1.1 - 2005.02.12 - Converted page to user control. Added option to specify the starting channel path.
  • V1.2 - 2006.05.14 - ASP.NET 2.0 version added.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Stephen Huen
Software Developer (Senior)
Canada Canada
A Microsoft Certified Professional Developer and Technology Specialist.
 
Experience and expertise in SharePoint 2013 / 2010 / 2007.
 
Role ranges from a developer in a multi-person team to a solution consultant with expert-level skills, leading a project to completion status.
 
Proven experience working effectively in a team environment and a self-managed environment.
Follow on   LinkedIn

Comments and Discussions

 
GeneralContent Search PinmemberRoopeshPerlaKumar3-Feb-06 1:06 
GeneralUsing Google Search Engine Pinmemberkanid4-Feb-05 1:55 
GeneralRe: Using Google Search Engine PinmemberStephen Huen7-Feb-05 22:04 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.150123.1 | Last Updated 17 May 2006
Article Copyright 2004 by Stephen Huen
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid