Click here to Skip to main content
Click here to Skip to main content

A Web Content Monitoring Tool: Web Watch - Gas Price

, 9 Jan 2006
Rate this:
Please Sign up or sign in to vote.
This is an article to demonstrate how to write a web content monitoring tool to watch gas price indices.

Sample Image - webwatch_img.gif

Introduction

Gas prices has soared up everywhere these last years. In order to find a gas station in my area to buy less expensive gas, I often visit www.TorontoGasPrices.com. This web site provides a place for drivers to report gas prices around the Great Toronto Area (GTA) in Ontario, Canada. While the gas prices at night are much cheaper than in the morning, some stations always have lower gas prices than others. I felt that I needed a tool to monitor the reported gas prices on that web site for me. Meanwhile, I'd just got a Visual Studio 2005 Standard version from Microsoft's 2005 Launch event and was trying to find a first project for it. So I started writing this tool myself. I have to say that I had lots of fun with this little project. Hope you enjoy using the tool as well.

The tool was written in C# on Microsoft .NET Framework 2.0. In order to install the software and source code, you should have Windows 2000/XP with Microsoft .NET Framework 2.0 installed. If you just want to use the tool (not the source), the Framework re-distributable package is all you need (Microsoft download site).

I know most of us still use VS 2003. For example, I'm planning to keep both VS 2003 and 2005 for a while. I apologize if you cannot load it with VS 2003.

The Requirements for the "Web Watch - Gas Price" Tool

I have defined the following requirements based on my needs:

  1. The tool will have a browser window for showing the web page we are monitoring.
  2. The user can select a city/area and location/gas station.
  3. The user can set a price to notify.
  4. When the gas price at the selected location has dropped below the mark, the tool notifies the user.
  5. The tool can be run with the window shown or hidden; when it's hidden, a notify icon on the taskbar systray provides an interface for the user to access the tool.

During development, I found that we could actually support more functions with a little effort. Thus, I added two more requirements:

  1. The user may use the browser to visit and navigate other web sites.
  2. The user may configure the behavior of the tool at next run, e.g., automatically hide the window and start a watch by using the previous settings.

The Tool User Manual

How to Set a Gas Price Watch

  1. Select a city/area on the "City/Area" combo box.
  2. Select a location/station on the "Location" combo box.
  3. Click the "Set Watch" menu to set a price to watch.
  4. Click the "Start" menu item to start monitoring. When the gas price at the selected location has dropped below the watch price, Web Watch will pop up a notice window with some music.
  5. Click the "Stop" menu item to stop monitoring.
  6. Click "Stand By" to hide the Web Watch window while the watch is monitoring.
  7. Click the "Clear->Current" menu item to clear the current window data and stop the watch if it is monitoring.
  8. Click the "Exit" menu to close the program.

How to Use the Web Watch Icon on the Systray

  1. Double clicking on the Web Watch icon on the systray will show the window.
  2. Right clicking on the Web Watch icon on the systray shall pop up the context menu with the following options:
    • Show/Hide WebWatch
    • Set Price
    • Start/Stop Monitoring
    • Close

These options are duplicates of the watch functionality, which will be used when the watch window is hidden.

How to Configure Web Watch for the Next Run

  1. If you want the Web Watch window to be shown (or hidden) when Web Watch is started the next time, select the "Show Window" (or "Hide Window") radio button.
  2. Web Watch saves all the session data in a database. When it starts, it will load all the settings from the last run (city/location/company/price to watch/etc.).
  3. When Web Watch is started with the window hidden, you may invoke Step 9 to show the window and set/change a price to watch.
  4. You may click the "Clear->History" menu item to clear out all the history data so that the next run will start from scratch.

"Battery Included" Features

  1. You may use Web Watch as a web browser. To visit a web page, enter the web address in the URL text box and press the "Enter" key.
  2. You may use the navigation menus "Previous" or "Next" to navigate web sites backward or forward.

The Tool Developer Manual

Beside the usefulness of the tool, the project itself has demonstrated the use of the following features for C# beginners:

  • Windows Forms controls: combo box, timer, text box, tool-menu strip, browser window, notify icon, icon menu, etc.
  • File I/O, string, stream reader/writer
  • Property as a means of inter-form communication
  • Double linked list as a collection type

To help yourself to read the source code, visit the web page www.TorontoGasPrices.com. On the browser window, click "View->Source" menu to get the HTML source. After Notepad has loaded the HTML source, make sure that the "View->Word Wrap" menu is unchecked. Leave the Notepad window open; you will need to revisit the HTML frequently when reviewing the C# source code.

There are three major segments of the C# source code:

  1. Web content request
  2. Search functions
  3. Event handlers

The web source request code requests web contents based on the URL. Then it stores the HTML source as string lines in a double linked list. I've chosen double linked list because we need to be able to search in either direction depending on the way the contents are organized. Here is the code of ReadHtmlSrc():

public void ReadHtmlSrc(string url)
{
    // Clear the html source list
    htmlSrc.Clear();

    try
    {
        // Create a request for the URL.
        WebRequest request = WebRequest.Create(url);
        // If required by the server, set the credentials.
        request.Credentials = CredentialCache.DefaultCredentials;

        // Get the response.
        HttpWebResponse response = (HttpWebResponse)request.GetResponse();

        // Get the stream containing content returned by the server.
        Stream dataStream = response.GetResponseStream();
        // Open the stream using a StreamReader for easy access.
        StreamReader reader = new StreamReader(dataStream);

        // Read and store the content.
        while (!reader.EndOfStream)
        {
            string oneLine = reader.ReadLine();
            htmlSrc.AddLast(oneLine);
        }

        // Clean up 
        reader.Close();
        dataStream.Close();
        response.Close();
    }
    catch (WebException caught)
    {
        MessageBox.Show(caught.ToString(), "Web Watch - Web Request Error");
        throw;
    }
}

Several search functions are designed to process user selections. As an example, the following function is used to query the available locations for a given city/area. As you can see, whenever the query finds a location, the function adds it to the "Location" combo box.

private void GetLocationList(string cityName, 
             ref System.Windows.Forms.ToolStripComboBox locations)
{
    Int32 index;
    Int32 offset = 22;
    Int32 tail = 5;
    string location = "";
    bool bCityFound = false;

    locations.Items.Clear();

    LinkedListNode<STRING> current = htmlSrc.First;

    while (current != null)
    {
        string line = current.Value;
        Int32 length = line.Length;

        if (!bCityFound)
        {
            if (line.IndexOf("PList") >= 0 && line.IndexOf(cityName) >= 0)
            {
                bCityFound = true;
            }                        
        }
        else
        {
            index = line.IndexOf("PAddress");
            if (index >= 0)
            {
                location = line.Substring(index + offset,
                                    length - index - offset - tail);
                
                if(!locations.Items.Contains(location))
                    locations.Items.Add(location);
                bCityFound = false;
            }
        }

        current = current.Next;
    }
}

The next function searches the gas company name for a given location. In this case, we start from the last item of the list, because the web contents appear in the reverse order in the HTML source.

private bool GetGasCompany(string stationName, ref string company)
{
    Int32 index;
    bool bGotCompany = false;
    company = "";

    if (stationName == "") return bGotCompany;

    bool bFoundLocation = false;

    // Since the order of appearance for price, gas company, location in the HTML
    // source may vary, depending on individual web sites, we store it in a doubly 
    // linked list so that the search order can be one way or another. In this case,
    // we want to start from location to gas company. We start the search on the
    // list from the last item.

    LinkedListNode<STRING> current = htmlSrc.Last;

    while (current != null)
    {
        string line = current.Value;
        Int32 length = line.Length;

        if (!bFoundLocation)
        {
            // Search for the location string
            index = line.IndexOf(stationName);

            if (index >= 0)
            {
                bFoundLocation = true;
            }
        }
        else
        {
            foreach (string comName in gasCompanies)
            {
                // Search for the company string
                index = line.IndexOf(comName);

                if (index >= 0)
                {
                    company = comName;
                    bGotCompany = true;
                    break;
                }
            }

            // Search for the string after the company string
            index = line.IndexOf("PStation");

            if (index >= 0 || bGotCompany)
            {
                break;
            }
        }

        current = current.Previous;
    }

    return bGotCompany;
}

A sound file "WebWatch.wav" and an icon file "WebWatch.ico" are required to be in the same folder as the executable, which is the Release folder in the current build.

Points of Interest

As you can see, the implementation of "Web Watch - Gas Price" is quite simple by all means. It has small footprints, yet performs what we expect it to do. I am quite happy with what the tool turned out to be. From a developer's point of view, however, there is a limitation to the implementation. It is bound to a specific web page. If one day the web page changes, the search code will be forced to change or it becomes useless.

The real issue appears to be that how to monitor a web page depends on the business context and the organization of the web content. When a criterion is given, how we search the web page for the target depends on two aspects:

  1. The clue selected for the target. It is important to pick an appropriate clue for the target based on the criterion. A better clue saves a lot of searching effort.
  2. The way the web page is organized. In the HTML source, the clue and/or criterion may appear before or after the target. The clue and target may appear in the same line or separate lines.

We may use some form of configuration that tells a search function what the clue is for the target and how to perform the search, but it does not solve the entire problem. I'd like to see if we could come up with better approaches to designing generic web monitoring engines in future updates of this article.

Conclusion

This article has presented a gas price monitoring tool, Web Watch - Gas Price, which monitors gas prices on www.TorontoGasPrices.com. Although the final product has exceeded the project requirements, its implementation is bound to a specific web page. The discussion on its implementation has raised a question: how do we write a generic search engine for the sake of web contents monitoring? I'm still searching for better and more generic web monitoring solutions.

History

This is the first revision of the article and source code.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Jun Du
Architect GuestLogix Inc.
Canada Canada
Jun is an experienced software architect. He wrote his first computer code on the tape machine for a "super computer". The tape machine reads holes on the black pape tape as source code. When manually fixing code, you need a punch and tranparent tape. To delete code, you block holes or cut off a segment and glue two ends together. To change code, you block old holes and punch new holes. You already know how to add new code, don't you? Anyway, that was his programming story in early 1980's.
 
Jun completed university with the specialty in oceanography, and graduate study in meteorology. He obtained his Ph.D. in physics. Jun has worked in a number of different areas. Since mid-90's, he has been working as a software professional in both military & commercial industries, including Visual Defence, Atlantis Systems International and Array Systems Computing.
 
Currently, Jun is an architect at GuestLogix, the global leader in providing onboard retail solutions for airlines and other travel industries. He is also the founder of Intribute Dynamics, a consulting firm specialized in software development. He has a personal blog site, although he is hardly able to keep it up to date.
 
In his spare time, Jun loves classic music, table tennis, and NBA games. During the summer, he enjoyes camping out to the north and fishing on wild lakes.

Comments and Discussions

 
Generalit has other interseting uses like kitesurfing Pinmemberjasperp21-Dec-05 0:56 
GeneralRe: it has other interseting uses like kitesurfing PinmemberJun Du22-Dec-05 3:44 
GeneralRe: it has other interseting uses like kitesurfing PinmemberGreat George Smith5-Jun-08 3:52 
Generalgreat article but useless to most people on 2003 PinmemberPetey Boy13-Dec-05 0:40 
GeneralRe: great article but useless to most people on 2003 PinmemberJun Du13-Dec-05 5:29 
GeneralJust an idea PinmemberStoyan Damov11-Dec-05 22:29 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140814.1 | Last Updated 9 Jan 2006
Article Copyright 2005 by Jun Du
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid