WebResourceProvider

• Download demo project - 78 Kb

Introduction

• search engines like Google and Altavista
• online stock quote providers such as Yahoo, CNN and Quicken
• weather information providers such as Weather.com, AccuWeather and CNN
• online translation services like Google, Babelfish and FreeTranslation
• etc.

The demo application included with this article shows how you can easily create objects to get:

• stock quotes
• the weather for a US zip code
• the list of locations served by a US zip code
• the translation of a piece of text
• the list of broken links on an HTML page
• the list of top posters at CodeProject

A Word of Caution

How it Works

WebResourceProvider works by initializing itself, constructing a URL to be retrieved, downloading the resource, and extracting useful information from the downloaded content. The process is repeated until no more data needs to be downloaded.

You use WebResourceProvider by deriving your own resource provider class from it, and overriding any of these virtual methods (shown in red in the diagram on the right):

• init
• constructUrl
• isPost()
• getPostData()
• parseContent()
• moreAvailable()

WebResourceProvider provides an assortment of methods to help parse downloaded content. They are:

Sample Resource Providers

Here are screenshots of the sample resource providers in action.

The QuoteProvider object works by posting a request to Yahoo's basic stock quote form and parsing the returned information.

The ZipCodeDecoder object works by posting a request to the USPS zip code association form and parsing the returned information.

The WeatherProvider object works by posting a request to CNN's weather form and parsing the returned information.

The Translator object works by posting a request to Google's translation engine and parsing the returned information. The request includes the translation mode. The sample performs a reverse translation and presents it along with the original text for comparison purposes.

The LinkChecker object is a thin layer above the WebResourceProvider class. It delegates the job of determining the document and image links at a URL to the base class. LinkChecker::getLinks() does its best to determine the links on a page, but because of the large number of ways a link can be specified, this method may miss a few links. The LinkChecker demo uses WebResourceProvider::urlExists() to check whether a link is valid. Click here to see a screenshot of the LinkChecker demo run against the CodeProject home page. Keep up the good work, Chris!

Using WebResourceProvider

1. Build the WebResourceProvider_Lib project.
2. Modify your application's project to look for header files and libraries in the WebResourceProvider_Lib project area.
3. Derive an object from WebResourceProvider. You'll need to #include WebResourceProvider.h in your derived class' header file.
4. Override the constructUrl() method in your derived class. This method specifies the URL to be downloaded.
5. Override the parse() method in your derived class. This method extracts information from the downloaded content and stores it in the derived class' member variables.
6. Optionally override other WebResourceProvider virtual methods. See the source code of the sample resource providers included in the demo project for examples.
7. Link your application with WebResourceProvider.lib.

Acknowledgement

• WebGrab by Chris Maunder
• AmHttpUtilities by Anders Molin
• Case-Insensitive String Replace by Uwe Keim

A Call for Interesting WebResourceProviders!

This is an invitation to the CP community to come up with interesting and useful web resource providers. Let your imagination (and coding prowess) flow! Please post your cool WebResourceProvider derived classes at CodeProject.

Revision History

• 23 Mar 2007
  Updated parsing logic in WeatherProvider, ZipCodeProvider and Translator modules.
• 7 Oct 2006
  Bug Fix: Updated parsing logic in Translator module.
• 7 Aug 2002
  Bug Fix: Fixed computation error in extractToEnd().
  Bug Fix: Added missing call to init() in fetchResource().
  Added methods skipBackTo() and skipBackToExact.
• 8 May 2002
  Added methods urlExists(), getLinks(), removeComments(), removeScripts(), findNoCase() and findStringInArray().
  Modified at(), skipTo(), and extractTo() to be case insensitive. Added case sensitive analogs atExact(), skipToExact() and extractToExact().
  Added LinkChecker sample to demo app.
  Fixed a bug in parse() that caused the fetch status to be ignored.
  Speeded up ZipCodeDecoder sample object.
• 30 April 2002
  Corrected control flow image.
• 29 April 2002
  Initial version of article.