Which technology to use for running crawler and updating database in back end ?

Question

5.00/5 (2 votes)

See more:

Hi
I am developing a project for college and i need some suggestions over the development. Its a website which shows information from other websites like Links, Images etc.

I have prepared below given model for the website.

A Home.aspx page which shows data from tables (sql server).

I have coded a crawler (in c#) which can crawl (fetch data) required website data.

I want some way through which i can run the crawler at back end for some time interval and it can insert updates in the tables. I want that i can get updated information in my database so that Home.aspx shows updated info. (Its like smaller version of Google News website)

I want to host the wesbite in Shared Hosted Environment (i.e a 3rd party hosting provider company and that may use IIS platform)

I posted simliar situation to different .NET forums and communities and they suggested lot of different things such as

- Create a web service (is it really necessary ?)

- Use WCF

- Create a Console application and run windows task sheduler (is it okay with asp.net (win forms website) and in shared hosted)

- Run crawler on local machine and update database accordingly. (No i want everything online)

Please suggest me a clear way out so that i complete the task. Please suggest elobrated technology and methods which suits my project.

Waiting...

Thanks...

Posted 14-Jan-12 3:24am

LockedOut

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

jinxster · Answer 1 · 2012-01-14T06:22:00

Solution 1

Depending on your hosted environment, most hosting companies allow you to make a cronjob for things like that. I don't know if IIS can do those type of things but on *nix boxes it is a lot more common occurance.

Posted 14-Jan-12 6:22am

jinxster

Comments

SvenMe 14-Jan-12 12:37pm

The counterpart of cron is the Windows Task Scheduler.

jinxster 14-Jan-12 20:35pm

Ok it is the same, in cron you would run the php script directly. So in Windows it would be to run the asp script. Much like your console app idea, but just make another asp file and schedule it to run. Somehow I think your task scheduler would need to be configured in a way to run the asp program to understand asp with the parameters of the file so it executed. I dont know if that is a simple as it looks, hence you got all the other suggestions.