Click here to Skip to main content
16,006,341 members
Home / Discussions / C#
   

C#

 
GeneralRe: Problem with "Cannot implicitly convert type "string" to "int" While using 2 buttons on Windows form application Pin
N a v a n e e t h18-Oct-09 18:02
N a v a n e e t h18-Oct-09 18:02 
GeneralRe: Problem with "Cannot implicitly convert type "string" to "int" While using 2 buttons on Windows form application Pin
WhiteWolf1918-Oct-09 18:04
WhiteWolf1918-Oct-09 18:04 
GeneralRe: Problem with "Cannot implicitly convert type "string" to "int" While using 2 buttons on Windows form application Pin
Luc Pattyn18-Oct-09 18:06
sitebuilderLuc Pattyn18-Oct-09 18:06 
GeneralRe: Problem with "Cannot implicitly convert type "string" to "int" While using 2 buttons on Windows form application Pin
N a v a n e e t h18-Oct-09 18:18
N a v a n e e t h18-Oct-09 18:18 
AnswerRe: Problem with "Cannot implicitly convert type "string" to "int" While using 2 buttons on Windows form application Pin
AhmedMasum19-Oct-09 0:44
AhmedMasum19-Oct-09 0:44 
QuestionWeb site Ripper (Copier) in C# Pin
fireblade8018-Oct-09 13:56
fireblade8018-Oct-09 13:56 
AnswerRe: Web site Ripper (Copier) in C# Pin
Christian Graus18-Oct-09 14:06
protectorChristian Graus18-Oct-09 14:06 
AnswerRe: Web site Ripper (Copier) in C# Pin
N a v a n e e t h18-Oct-09 17:08
N a v a n e e t h18-Oct-09 17:08 
As CG said keeping track of the links already visited and avoid duplicate requests would be challenging as links can appear in different forms like http://somewebsite.com or http://www.somewebsite.com. Other area where you will have problems will be to handle links that uses JS. It won't be simple hyperlinks but calls a JS function and redirects to a page.

fireblade80 wrote:
Is there anything specific that I should be careful with?


There are few general things to consider.

  1. Use a data structure that offers fast lookups for keeping the visited URLs. Using sequential data structures like List(T) will be a poor choice. Try with HashSet(T) which uses hash tables internally and gives you lookups in constant time.
  2. You don't really need to construct the link tree. A better approach would be to parse the links immediately after finding. I mean a producer/consumer queue would be good. A producer thread finds all the links and put it to a queue. Several consumer threads will be listening to the queue and processing each links. This way you can save the time you use for constructing link tree
  3. Some pages will return a HTTP status code 301 or 302 which means Moved Permanently and 302 Found. Your code should be able to handle this and request again to the changed URL.
  4. Define a depth level so that your application will not go into too depth when grabing URLs
Smile | :)

Best wishes,
Navaneeth

GeneralRe: Web site Ripper (Copier) in C# Pin
Luc Pattyn18-Oct-09 18:11
sitebuilderLuc Pattyn18-Oct-09 18:11 
GeneralRe: Web site Ripper (Copier) in C# Pin
N a v a n e e t h18-Oct-09 18:20
N a v a n e e t h18-Oct-09 18:20 
QuestionCannot set listview colum data correctly Pin
c#coder200918-Oct-09 13:34
c#coder200918-Oct-09 13:34 
AnswerRe: Cannot set listview colum data correctly Pin
N a v a n e e t h18-Oct-09 17:25
N a v a n e e t h18-Oct-09 17:25 
GeneralRe: Cannot set listview colum data correctly Pin
c#coder200918-Oct-09 18:06
c#coder200918-Oct-09 18:06 
GeneralRe: Cannot set listview colum data correctly Pin
N a v a n e e t h18-Oct-09 18:11
N a v a n e e t h18-Oct-09 18:11 
GeneralRe: Cannot set listview colum data correctly [modified] Pin
c#coder200918-Oct-09 18:25
c#coder200918-Oct-09 18:25 
GeneralRe: Cannot set listview colum data correctly Pin
N a v a n e e t h19-Oct-09 5:35
N a v a n e e t h19-Oct-09 5:35 
Questionborzoi 1.02 and C# Pin
adam syria18-Oct-09 11:41
adam syria18-Oct-09 11:41 
AnswerRe: borzoi 1.02 and C# Pin
Christian Graus18-Oct-09 14:07
protectorChristian Graus18-Oct-09 14:07 
QuestionWCF - Client Wait Other Client [modified] Pin
dataminers18-Oct-09 11:13
dataminers18-Oct-09 11:13 
AnswerRe: WCF - Client Wait Other Client Pin
Ravi Bhavnani18-Oct-09 18:11
professionalRavi Bhavnani18-Oct-09 18:11 
GeneralRe: WCF - Client Wait Other Client Pin
dataminers18-Oct-09 20:53
dataminers18-Oct-09 20:53 
AnswerRe: WCF - Client Wait Other Client [modified] Pin
Mirko198018-Oct-09 23:08
Mirko198018-Oct-09 23:08 
Questioncustom control events Pin
teknolog12318-Oct-09 10:00
teknolog12318-Oct-09 10:00 
AnswerRe: custom control events Pin
DaveyM6918-Oct-09 10:22
professionalDaveyM6918-Oct-09 10:22 
GeneralRe: custom control events Pin
teknolog12318-Oct-09 12:01
teknolog12318-Oct-09 12:01 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.