WebResourceProvider goes .NET

• Download source - 5 Kb
• Download documentation - 22 Kb

Introduction

This article (an improved .NET version of my C++ version) describes WebResourceProvider, a simple yet powerful framework for retrieving useful information from public sources on the web, such as:

• search engines like Google and Altavista
• online stock quote providers such as Yahoo, CNN and Quicken
• weather information providers such as Weather.com, AccuWeather and CNN
• online translation services like Google, Babelfish and FreeTranslation
• RSS feeds like Yahoo World News, CNN Technology News and Google Business News,
• etc.

Apart from being rewritten in C# from the ground up, this .NET version offers a smaller footprint than its C++ predecessor and makes it easy (almost trivial) to encapsulate functionality offered by online services into an object that can be manipulated by your application. Here are screenshots of some WebResourceProvider applications (listed at the end of this article) in action.

Domain Walker

Google Translator

Simple RSS Reader

A Word of Caution

Before you use WebResourceProvider to write the next killer app, be aware that there are legal and ethical issues regarding the use of information obtained from other sources. In particular, the terms of service (TOS) of content providers such as Yahoo, CNN, etc. clearly state what you can and cannot do with information retrieved from their sites. Even if you write a web resource provider for personal use only, you should take into consideration any undue stress that your object may put on a web server. The object's Pause property allows you to inject a delay between successive HTTP requests to help prevent overloading a server.

How it Works

WebResourceProvider works by initializing itself, constructing a URL to be retrieved, downloading the resource, and extracting useful information from the downloaded content. This process repeats until no more data needs to be downloaded. You use WebResourceProvider by deriving from it, overriding getFetchUrl() and optionally overriding any of these virtual methods (shown in red in the flowchart on the right): init() getPostData() parseContent() continueFetching() In the spirit of true object orientation, WebResourceProvider (unlike its C++ predecessor) doesn't provide a facility for parsing downloaded content. Readers are instead urged to use my StringParser class to help perform this task. WebResourceProvider exposes the following properties: Agent Gets and sets the user agent string. Content Gets the retrieved content. ErrorMsg Gets the last error message, if any. FetchTime Gets the fetch timestamp. Pause Gets and sets the minimum pause time interval (in mSec). Referer Gets and sets the referer string. Timeout Gets and sets the timeout (in mSec).

Demo applications

C# applications (with full source code) that use WebResourceProvider can be found here:

• DomainWalker - a web topology analyzer
• GoogleTranslator - an object that uses Google to translate natural language
• SimpleRSS - an RSS channel reader

Revision History

• 17 Feb 2008
  Updated links to sample applications.

• 15 Jan 2006
  Initial version.