Hi, I have tried to get a starting on making a web crawler. Was progressing well till I got this confusion That i can't understand. I have written the following code:
I am passing "http://www.google.com" as the string "URL"
public void crawlURL(string URL, string depth)
{
if (!checkPageHasBeenCrawled(URL))
{
PageContent = getURLContent(URL);
MatchCollection matches = Regex.Matches(PageContent, "href=\"", RegexOptions.IgnoreCase);
int count = matches.Count;
}
}
private string getURLContent(string URL)
{
string content;
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(URL);
request.UserAgent = "Fetching contents Data";
WebResponse response = request.GetResponse();
Stream stream = response.GetResponseStream();
StreamReader reader = new StreamReader(stream);
content = reader.ReadToEnd();
reader.Close();
stream.Close();
return content;
}
Problem:
I am trying to get all the Links of the page(http://www.google.com) but I see less count of the links from the Regex matches. It gives me links count to be 19 while when i checked the source code manually for the word "href=" it gave me 41 occurances. I can't understand why it is giving me less count of the word from the code.
Also One other problem I am currently facing is how to make sure that the links I have obtained are not broken links. They should map to some place. One way I can think of is the same way I am getting www.google.com page content by httpwebrequest object. Is thier another way or this would be the best. Thanks !!
[Edit]Code block added[/Edit]