Beginner's question: create a "crawler"?

Question

1.00/5 (1 vote)

See more: (untagged)

hi there,
i've done a bunch of googling, but maybe i'm just using the wrong word. (not sure if it's actually called a "crawler", what I'd like to do)

here in germany there are several sites, which post travel deals, like [spam removed]

what i'd like to do is build a tool to "search" (or crawl?!) these websites and pick out only the deals (in this case "articles" because the websites just show articles/posts that link further) that pertain to departure airport in Berlin, and show these on a website.

is it a crawler that i'm looking to build, or how would one call such?

thanks a lot!
best,
zac

Posted 12-Jun-14 21:08pm

Member 10881943

Updated 12-Jun-14 21:13pm

Thanks7872

v2

Add a Solution

Comments

Thanks7872 13-Jun-14 3:13am

Don't mention site names like this. It is considered Spam here on Codeproject.com

Member 10881943 13-Jun-14 7:56am

sorry! wasn't meant that way.

Richard MacCutchan 13-Jun-14 3:40am

Yes, a web crawler. So what exactly is your question?

Member 10881943 13-Jun-14 7:56am

Oh, spam? didn't know that. not meant to be!
Well, my question was more or less that, because when I was looking for "Tutorials" or so for crawlers (because i thought it was called a crawler but wasn't 100% sure) they were not really helping me further along.

Anyone know a good place to learn how to build one?!

Richard MacCutchan 13-Jun-14 9:06am

Lots of things to learn, not all totally connected. You will need to do a lot of research, and learn how to 'find' things on the internet. There are plenty of articles here on CodeProject about the technical bits, but be prepared for a lot of hard work.

NeverJustHere 15-Jun-14 2:58am

You will also have to read he terms of use of the websites you are scanning to make sure you are allowed to. There are many sites that get annoyed by this type of thing, and will actively look to detect and block you.

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)