Click here to Skip to main content
12,827,305 members (42,601 online)
Click here to Skip to main content
Add your own
alternative version


21 bookmarked
Posted 13 Dec 2007

Google Site Map Crawler

, 13 Dec 2007 CPOL
Rate this:
Please Sign up or sign in to vote.
Console application that chacks all URLs listed in sitemap.xml file


Have you ever thought of trying to validate each URL listed in your sitemap file?<o:p />


I have a site with dynamically generated page links. Those links are generated based on a page title which can be any combination of letters, numbers and symbols. Of course, the site does remove all forbidden characters from the page title before generating its URL, trims and shortens it a bit... however errors still occur from time to time. For example, a page with a title: ''...IS_BROKEN'' ''' due to my URL conversion specifics will have the following URL: /.IS_BROKEN+ There are thousands of pages so it’s clear that I can not verify each separate page that the site’s database contains.<o:p />

Based on a list of dynamically generated URLs I generate a sitemap.xml file. Which contains all of the site pages. So each time a map-file is generated I need to ensure that there are no repeating items (this may happen if different pages have same titles) and each separate URL is accessible, i.e. does not produce either bad request, or 404 or anything like that.<o:p />

<o:p />So I created a C# program that walks through each URL listed in the sitemap.xml file and tries to access it. It logs all errors occurred into an output file, so it’s easy to track problem pages.

I use XmlDocument class for loading a sitemap.xml; WebRequest and WebResponse classes for determination of whether a URL exists.


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Ukraine Ukraine
No Biography provided

You may also be interested in...

Comments and Discussions

GeneralMy vote of 2 Pin
gg6715-Dec-09 9:44
membergg6715-Dec-09 9:44 
GeneralMinor spelling in article sub-title Pin
Tatworth14-Dec-08 6:24
memberTatworth14-Dec-08 6:24 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.170326.1 | Last Updated 13 Dec 2007
Article Copyright 2007 by Summer_son
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid