Click here to Skip to main content
Click here to Skip to main content


, 23 Mar 2007 CPOL
Rate this:
Please Sign up or sign in to vote.
A framework to allow public web services to be used as objects in your application.


This article describes WebResourceProvider, a simple yet powerful framework for retrieving useful information from publicly available web services. I use the term "web service" in a generic, non-Microsoft sense, to mean information providers such as:

The demo application included with this article shows how you can easily create objects to get:

  • stock quotes
  • the weather for a US zip code
  • the list of locations served by a US zip code
  • the translation of a piece of text
  • the list of broken links on an HTML page
  • the list of top posters at CodeProject

CodeProjectTopPosters object in action

A Word of Caution

Before you use WebResourceProvider to write the next killer app, be aware that there are legal and ethical issues regarding the use of information obtained from other sources. In particular, the terms of service (TOS) of content providers such as Yahoo, CNN, etc. clearly state what you can and cannot do with information retrieved from their sites. Even if you write a web resource provider for personal use only, you should take into consideration any undue stress that your object may put on a web server. The CodeProjectTopPosters example in the demo won't let you get at more than the top 40 CodeProject posters. Further, it pauses between multiple accesses to the CodeProject server in order to not overload it.

How it Works

WebResourceProvider control flowWebResourceProvider works by initializing itself, constructing a URL to be retrieved, downloading the resource, and extracting useful information from the downloaded content. The process is repeated until no more data needs to be downloaded.

You use WebResourceProvider by deriving your own resource provider class from it, and overriding any of these virtual methods (shown in red in the diagram on the right):

  • init
  • constructUrl
  • isPost()
  • getPostData()
  • parseContent()
  • moreAvailable()

WebResourceProvider provides an assortment of methods to help parse downloaded content. They are:

Method Purpose
at Checks whether current location is at a string
atExact Case sensitive version of at()
skipTo Advances current location to next occurence of a string
skipToExact Case sensitive version of skipTo()
skipBackTo Retreats current location to previous occurence of a string
skipBackToExact Case sensitive version of skipBackTo()
extractTo Extracts text from current location to the start of a string
extractToExact Case sensitive version of extractTo()
extractToEnd Extracts text from current location to end of content
getIndex Returns current location
getLinks Returns HREF and IMG links in content
resetIndex Sets current location to start of content
replaceEvery Replaces every occurence of a string in content with another
removeComments Removes comments from content
removeScripts Removes scripts from content
removeEnclosingAnchorTag Removes anchor tag enclosing a string
removeEnclosingQuotes Removes quotes enclosing a string
removeHtml Removes HTML from a string
trim Removes leading and trailing whitespace from a string

Sample Resource Providers

Here are screenshots of the sample resource providers in action.

QuoteProvider object in action The QuoteProvider object works by posting a request to Yahoo's basic stock quote form and parsing the returned information.

ZipCodeDecoder object in action The ZipCodeDecoder object works by posting a request to the USPS zip code association form and parsing the returned information.

WeatherProvider object in action The WeatherProvider object works by posting a request to CNN's weather form and parsing the returned information.

Translator object in action The Translator object works by posting a request to Google's translation engine and parsing the returned information. The request includes the translation mode.

The sample performs a reverse translation and presents it along with the original text for comparison purposes.

LinkChecker object in action The LinkChecker object is a thin layer above the WebResourceProvider class. It delegates the job of determining the document and image links at a URL to the base class.

LinkChecker::getLinks() does its best to determine the links on a page, but because of the large number of ways a link can be specified, this method may miss a few links.

The LinkChecker demo uses WebResourceProvider::urlExists() to check whether a link is valid.

Click here to see a screenshot of the LinkChecker demo run against the CodeProject home page. Keep up the good work, Chris!

Using WebResourceProvider

To use WebResourceProvider do the following:
  1. Build the WebResourceProvider_Lib project.
  2. Modify your application's project to look for header files and libraries in the WebResourceProvider_Lib project area.
  3. Derive an object from WebResourceProvider. You'll need to #include WebResourceProvider.h in your derived class' header file.
  4. Override the constructUrl() method in your derived class. This method specifies the URL to be downloaded.
  5. Override the parse() method in your derived class. This method extracts information from the downloaded content and stores it in the derived class' member variables.
  6. Optionally override other WebResourceProvider virtual methods. See the source code of the sample resource providers included in the demo project for examples.
  7. Link your application with WebResourceProvider.lib.


WebResourceProvider uses the following code written by others:

A Call for Interesting WebResourceProviders!

This is an invitation to the CP community to come up with interesting and useful web resource providers. Let your imagination (and coding prowess) flow! Please post your cool WebResourceProvider derived classes at CodeProject.

Revision History

  • 23 Mar 2007
    Updated parsing logic in WeatherProvider, ZipCodeProvider and Translator modules.
  • 7 Oct 2006
    Bug Fix: Updated parsing logic in Translator module.
  • 7 Aug 2002
    Bug Fix: Fixed computation error in extractToEnd().
    Bug Fix: Added missing call to init() in fetchResource().
    Added methods skipBackTo() and skipBackToExact.
  • 8 May 2002
    Added methods urlExists(), getLinks(), removeComments(), removeScripts(), findNoCase() and findStringInArray().
    Modified at(), skipTo(), and extractTo() to be case insensitive. Added case sensitive analogs atExact(), skipToExact() and extractToExact().
    Added LinkChecker sample to demo app.
    Fixed a bug in parse() that caused the fetch status to be ignored.
    Speeded up ZipCodeDecoder sample object.
  • 30 April 2002
    Corrected control flow image.
  • 29 April 2002
    Initial version of article.


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Ravi Bhavnani
Technical Lead
Canada Canada
Ravi Bhavnani is an ardent fan of Microsoft technologies who loves building Windows apps, especially PIMs, system utilities, and things that go bump on the Internet. During his career, Ravi has developed expert systems, desktop imaging apps, marketing automation software, EDA tools, a platform to help people find, analyze and understand information, trading software for institutional investors and advanced data visualization solutions. He currently works for a company that provides enterprise workforce management solutions to large clients.
His interests include the .NET framework, reasoning systems, financial analysis and algorithmic trading, NLP, HCI and UI design. Ravi holds a BS in Physics and Math and an MS in Computer Science and was a Microsoft MVP (C++ and C# in 2006 and 2007). He is also the co-inventor of 2 patents on software security and generating data visualization dashboards. His claim to fame is that he crafted CodeProject's "joke" forum post icon.
Ravi's biggest fear is that one day he might actually get a life, although the chances of that happening seem extremely remote.
Follow on   Google+   LinkedIn

Comments and Discussions

GeneralNeeds to fill in textboxes and push a button Pinmembermethhoo27-Jan-08 14:08 
GeneralRe: Needs to fill in textboxes and push a button Pinmembermethhoo30-Jan-08 15:01 
GeneralRe: Needs to fill in textboxes and push a button PinmemberRavi Bhavnani30-Jan-08 18:12 
GeneralRe: Needs to fill in textboxes and push a button Pinmembermethhoo6-Feb-08 7:26 
GeneralRe: Needs to fill in textboxes and push a button PinmemberRavi Bhavnani8-Apr-08 4:18 
QuestionHow to read Yahoo inbox? Pinmemberpandian narayanan24-Jul-07 21:52 
AnswerRe: How to read Yahoo inbox? PinmemberRavi Bhavnani25-Jul-07 4:06 
QuestionAutomating posting to Google translator PinmemberAkhil Bhandari12-Jun-07 7:31 
AnswerRe: Automating posting to Google translator PinmemberRavi Bhavnani12-Jun-07 7:56 
GeneralTranslator problems Pinmemberhitesh_sharma23-Mar-07 3:08 
AnswerRe: Translator problems PinmemberRavi Bhavnani23-Mar-07 7:03 
GeneralRe: Translator problems Pinmemberhitesh_sharma23-Mar-07 8:57 
Generalgoogle's translate can't work again Pinmemberguilechao27-Feb-07 1:56 
GeneralRe: google's translate can't work again PinmemberRavi Bhavnani27-Feb-07 2:50 
GeneralRe: google's translate can't work again Pinmemberhitesh_sharma20-Mar-07 9:10 
AnswerRe: google's translate can't work again PinmemberRavi Bhavnani23-Mar-07 7:02 
AnswerRe: google's translate can't work again PinmemberRavi Bhavnani23-Mar-07 7:03 
QuestionIs there something glaringly obvious I'm missing? PinmemberDiarrhio9-Oct-06 8:37 
AnswerRe: Is there something glaringly obvious I'm missing? PinmemberRavi Bhavnani9-Oct-06 10:04 
Questionhow to use vc++ to control the CPU usage rate? Pinmemberiiswan9-Aug-06 9:07 
QuestionGoogle Translator doesn't work [modified] Pinmembernav_karla8-Aug-06 19:37 
AnswerRe: Google Translator doesn't work PinmemberRavi Bhavnani9-Aug-06 3:08 
GeneralRe: Google Translator doesn't work Pinmembernav_karla9-Aug-06 8:54 
AnswerArticle updated PinmemberRavi Bhavnani7-Oct-06 7:07 
QuestionHow can we change encoding ? PinmemberSygrael13-Feb-06 9:54 
AnswerRe: How can we change encoding ? PinmemberRavi Bhavnani13-Feb-06 10:49 
GeneralRe: How can we change encoding ? PinmemberSygrael14-Feb-06 12:04 
GeneralRe: How can we change encoding ? PinmemberSygrael6-Mar-06 9:59 
QuestionUnicode??????? PinmemberTcpip200531-Aug-05 3:56 
GeneralLets mantain this tool! Pinmembermirso6722-Jul-05 0:53 
QuestionHow many Pages r Checked? Pinmembermirso6720-Jul-05 6:44 
AnswerRe: How many Pages r Checked? PinmemberRavi Bhavnani20-Jul-05 7:47 
GeneralRe: How many Pages r Checked? Pinmembermirso6720-Jul-05 9:38 
GeneralAbsolutely fan-tab-ulous PinmemberTom Archer9-Feb-05 18:13 
GeneralRe: Absolutely fan-tab-ulous PinmemberRavi Bhavnani9-Feb-05 18:20 
GeneralRe: Absolutely fan-tab-ulous Pinmemberharishon4-Mar-05 13:04 
GeneralRe: Absolutely fan-tab-ulous PinmemberRavi Bhavnani29-Mar-05 4:03 
QuestionProxy Authentication??? PinmemberRafkat26-Nov-04 3:31 
AnswerRe: Proxy Authentication??? PinmemberRavi Bhavnani26-Nov-04 18:51 
GeneralC# PinmemberBrian Delahunty29-Sep-04 1:41 
GeneralRe: C# PinmemberRavi Bhavnani29-Sep-04 3:35 
GeneralRe: C# PinmemberBrian Delahunty29-Sep-04 3:58 
GeneralRe: C# PinmemberRavi Bhavnani29-Sep-04 4:04 
GeneralRe: C# PinmemberTom Archer9-Feb-05 18:14 
GeneralRe: C# PinmemberBrian Delahunty9-Feb-05 23:41 
Generalweather provider Pinmemberandyxp14-Feb-04 18:53 
GeneralRe: weather provider PinmemberRavi Bhavnani14-Feb-04 19:02 
GeneralRe: weather provider Pinmemberandyxp14-Feb-04 20:31 
GeneralRe: weather provider Pinmemberandyxp14-Feb-04 20:32 
GeneralDownload PinmemberLotusShiv20-Oct-03 9:51 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.150327.1 | Last Updated 23 Mar 2007
Article Copyright 2002 by Ravi Bhavnani
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid