Click here to Skip to main content
65,938 articles
CodeProject is changing. Read more.
Articles
(untagged)

WebResourceProvider goes .NET

0.00/5 (No votes)
17 Feb 2008 1  
A a simple yet powerful framework for retrieving useful information from public sources on the web.

Introduction

This article (an improved .NET version of my C++ version) describes WebResourceProvider, a simple yet powerful framework for retrieving useful information from public sources on the web, such as:

Apart from being rewritten in C# from the ground up, this .NET version offers a smaller footprint than its C++ predecessor and makes it easy (almost trivial) to encapsulate functionality offered by online services into an object that can be manipulated by your application. Here are screenshots of some WebResourceProvider applications (listed at the end of this article) in action.

A domain walker that discovers the topology of the world wide web.
Domain Walker
An object that encapsulates Google's online natural language translation tools.
Google Translator
RSSChannel and RSSItem - a pair of objects that allow you to build an RSS reader.
Simple RSS Reader

A Word of Caution

Before you use WebResourceProvider to write the next killer app, be aware that there are legal and ethical issues regarding the use of information obtained from other sources. In particular, the terms of service (TOS) of content providers such as Yahoo, CNN, etc. clearly state what you can and cannot do with information retrieved from their sites. Even if you write a web resource provider for personal use only, you should take into consideration any undue stress that your object may put on a web server. The object's Pause property allows you to inject a delay between successive HTTP requests to help prevent overloading a server.

How it Works

WebResourceProvider works by initializing itself, constructing a URL to be retrieved, downloading the resource, and extracting useful information from the downloaded content. This process repeats until no more data needs to be downloaded.

You use WebResourceProvider by deriving from it, overriding getFetchUrl() and optionally overriding any of these virtual methods (shown in red in the flowchart on the right):

  • init()
  • getPostData()
  • parseContent()
  • continueFetching()

In the spirit of true object orientation, WebResourceProvider (unlike its C++ predecessor) doesn't provide a facility for parsing downloaded content. Readers are instead urged to use my StringParser class to help perform this task.

WebResourceProvider exposes the following properties:

Agent Gets and sets the user agent string.
Content Gets the retrieved content.
ErrorMsg Gets the last error message, if any.
FetchTime Gets the fetch timestamp.
Pause Gets and sets the minimum pause time interval (in mSec).
Referer Gets and sets the referer string.
Timeout Gets and sets the timeout (in mSec).

WebResourceProvider control flow

Demo applications

C# applications (with full source code) that use WebResourceProvider can be found here:

Revision History

  • 17 Feb 2008
    Updated links to sample applications.
  • 15 Jan 2006
    Initial version.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here