29 Mar 2015 by Kornfeld Eliyahu Peter
There are two options:1. 'Crawling' the site - it means that you load the site's home page and follow every link on that page and load them and follow every link on them and so on...2. If the site supports the - very new and unsupported - sitemap-protocol, you can use...