How do I get a list of all webpages on a website in vb.net?

Question

0.00/5 (No votes)

See more:

How do I get a list of all webpages on a website in vb.net? I have tried lots of things - they return one error. I discarded all of them which I now regret.

Please help.

P.S. I do not want to make an xml sitemap.

Posted 19-Mar-15 1:56am

iProgramIt

Add a Solution

Comments

ZurdoDev 19-Mar-15 8:04am

What for?

You'd have to write a program like the spider search bots that search engines use. Not easy to do.

iProgramIt 19-Mar-15 8:07am

I know. I just want to do it for a school project.

ZurdoDev 19-Mar-15 8:11am

That's quite ambitious. You have to first connect to the website using no page, for example: codeproject.com and then parse all the html you get back. You have to find every link in that html and determine if it is a link within that site and if so you then have to write code to go to that url and do the same thing recursively until you've found every link on the site.

That still doesn't guarantee you found every page, there is no way of doing that.

iProgramIt 19-Mar-15 8:19am

Hmm, yes. That would also grab most of my computer's memory. Could I do a sitemap and remove all of the xml aspects?

ZurdoDev 19-Mar-15 8:23am

A sitemap is for when you have the website, when you already know all of the pages.

A spider app is when you are trying to get a list of all pages from a site you don't have the code for.

I'm still not sure what you are wanting.

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Kornfeld Eliyahu Peter · Answer 1 · 2015-03-19T06:08:00

It depends on the site...
There is a sitemap protocol[^], that is supported by the site it enables you to 'crawl' the site and get information of all the pages...
In any case writing such a crawler is not that simple, so you better search for some ready-made solution (there are free and open source solutions too, so do not worry about paying for :-))