Web Services For Ecologists

Download WebServiceProject.zip - 29.4 KB

Introduction

This article describes the creation of an ATL Web Service (Visual Studio 2005 IDE) and a Web Service client (ASP.NET). The ultimate goal is to describe a simple, efficient and repeatable methodology to integrate legacy C++ code (mathematical models and analyses) into a web service so that these models can be implemented (with minimal changes to the model code) via any web browser. By legacy code, we mean code that has been written some time ago, but that is still useful and that we do not wish to recode in another language.

Ecology and the World Wide Web

Ecology (the study of the interaction between plants and animals) is becoming an increasingly quantitative discipline. Present day ecologists are just as likely to work behind a computer as they are to romp around the countryside looking at plants and animals. Much of this work involves creating mathematical models and the software to implement them.

Ecological modeling has been successful enough that many of these models are used to support practical decision making for economically and sociologically important ecological problems. For example, decision support tools for managing the spread of harmful invasive species, documenting the location and status of endangered species and for identifying plant and animal species.

The general process of building models, implementing them into a software product and deploying them to multiple end users has a number of common issues that the technologies used in this article address:

1) Often models are a result of many years of ‘on and off’ funding. As a result people who built models and/or implemented code often leave the project. Although models should be well documented, it is often expensive or impractical to recreate, recode and revalidate them. Solution: Find methods that can integrate legacy code with modern deployment technologies.

2) Decision support tools are often a ‘suite’ of integrated models or analyses. For example, the first step in an ‘analysis’ may be to obtain data needed to perform an analysis by querying public and/or private data sources. This data may then be used to run one model. Then, the results of the first model might be consumed by a second model. Each model might be implemented as stand-alone program or a set of open source classes or functions. Solution: Develop software capable of integrating different models and that allows them to be called from the same GUI’s using standardized interfaces.

3) Research ecologists increasingly work as parts of larger teams. As in most teams, each member has particular strengths, weaknesses and responsibilities. This is particularly applicable to programming skills. While many ecologists are proficient in one or more computer languages, they are usually not computer scientists. Additionally, ecologists often work on multiple projects or teams doing the same sort of work but for different ecological applications. The same analyses (for example looking up specific types of data) are often common to a number of projects. Solution: Use a modular approach. Divide the project team into people responsible for writing analyses (with further subdivisions based upon ecological expertise) and those responsible for the GUI’s that the end-users ultimately interact with.

4) The success of many ecological models is driven not by how good or predictive they are, rather by who uses them, how easy they are to use and how much decision support they provide. Project funding is usually commensurate with the size of an ecological problem, so well funded projects may have large numbers of stake-holders (end users) eager to use the decision support tool. End users may be distributed all over the world, may use a variety of operating systems (often with usage restrictions) and have different levels of ecological expertise and computing experience. Solution: Use the WWW. Make the end-product easy to use (standard look and feel) and easy to upgrade and maintain.

With these problems and solutions in mind we decided to explore the potential of an ATL Web-Service consumed by C# web-browser code. Since most of our ecological models are written in unmanaged C++, we decided to use a Microsoft Visual Studio, ATL Web-Service project so that we did not have to worry about mixing managed and unmanaged code. For the client application we used C# and ASP.NET. We base the success of our investigations on:

1) Can we get it to work?

2) The ease with which the web-service can be built by a non-expert coder (plus any limitations to what it can do).

3) The ease with which an independent, non-expert coder (with minimal interaction from the web-service coder) can implement the web-service as a client web-page in Internet Explorer and Mozilla browsers.

To investigate this process, we will imagine that we have a legacy C++ coded ecological model (in this case a simple random number generator) written as a C++ class (or collection of classes) for which we have C++ code. For the purposes of this article, our pretend ‘model’ is trivial and simply returns a string of scaled random numbers. In this case, the outputs of the model could easily be achieved using C# and ASP.NET alone (so please don’t comment on this fact!) but it is relatively easy to image how, with more complex models this wouldn’t be quite so practical.

Using the code

This section will provide a walk-through for how to create a Web-Service, Deploy the Web Service and create a client that calls the Web Service.

Step 1: Coding the Web Service Server Application

1) Create a new ATL Server Web Service called “RandomNumberService”. Visual Studio will create 2 new Projects:

a. RandomNumberService

b. RandomNumberServiceISAPI

2) In the RandomNumberService project, expand the source and header folders and navigate to RandomNumberService.h. This file contains skeleton code that can be changed in order to customize the Web Service. The auto generated code from Visual Studio will contain the HelloWorld(..) as an example SOAP function. The declaration of the function is here:

// HelloWorld is a sample ATL Server web service
method. It shows how to// declare a web service method and its in parameters
and out-parameters 
[id(1)] HRESULT HelloWorld([in] BSTR bstrInput, [out, retval]
BSTR *bstrOutput); 
// TODO: Add additional
web service methods here 

and the implementation can be found in the implementation of the class CRandomNumberServiceService here:

// This is a sample web service method that shows how to use the  // soap_method attribute to expose a method as a web method
[ soap_method ]HRESULT HelloWorld(/*[in]*/
BSTR bstrInput, /*[out, retval]*/ BSTR *bstrOutput)
{ 
    CComBSTR bstrOut(L"Hello ");
    bstrOut += bstrInput;bstrOut += L"!";
    *bstrOutput = bstrOut.Detach();
    return S_OK;
}
// TODO: Add additionalweb service methods here

3) Imagine we have a C++ class (or a number of classes) that represent a model that we want to distribute (make available) across the web. In this case we are using the class CRandomNumbers as a simple, trivial example of a model (or any other piece of code) that might have been developed independently of the web service. This class contains a function GenRandomNumbers(..) which generates a series of uniform random numbers and returns them as a tab delineated string (it also contains another function GetGraph(..) which should be ignored for now):

CString GenRandomNumbers(double min, double max, int N)  

4) Follow the Visual Studio generated code comments and declare another Web Service method, GenRandomNumbers(..) below HelloWorld(). In this case, we want a function that mirrors the function declared in our ‘model’ code. So the Web Service method declaration is:

[id(2)] HRESULT GenRandomNumbers([in] double min,[in]double, max, [in] int N,[out, retval] BSTR *bstrOutput); 

and the implementation is:

[ soap_method ]
HRESULT GenRandomNumbers(double min,double max, int N,  BSTR *bstrOutput)
{
    ////Function that calls a function from an 'imaginarily' complex class and stroes the results in a string
    CRandomNumbers random;
    CString randString;
    randString = random.GenRandomNumbers(min, max, N);
    /////randString now contains the string of random numbers
    CComBSTR bstrOut(randString);
    *bstrOutput = bstrOut.Detach();
    return S_OK;
} 

5) The Web-Service now contains a method that calls an external class that returns a string of random numbers as a CString. The contents of this CString are then converted and returned to a BSTR which is wrapped up in the SOAP message and sent to any client program that calls GenRandomNumbers(…).

6) There is one important final step before building and deploying the Web-Service – that is to make some changes to “Visual Studio’s ATL Server Projects” auto-generated code that handles the request for a WSDL document. This WSDL document describes the methods exposed by the Web-Service server and makes building a Web-Service client easy. However, the auto-generated code produced by Visual Studio creates the document in a different format to the www SOAP standards. The following line of code solves this problem:

request_handler(name="Default",sdl="GenWebserverWSDL"),
soap_handler(name="RandomNumberServiceService",namespace="urn:RandomNumberServiceService",
protocol="soap",
style = "document",//add this argument
use = "literal" //add this argument) 

7) Now build the project. This will create 2 *.dll files: RandomNumberService.dll and RandomNumberServiceIsapi.dll. Move to Step 2 to deploy the Web Service server.

Step 2: Deploying the Web Service server

1) Install IIS on the machine you wish to deploy the Web Service.

2) Create a new virtual directory (in this case RandomGenerator) that will contain your Web Service and copy across the following files:

a. RandomNumberService.dll and RandomNumberServiceIsapi.dll located in the Release or Debug directory of the project

b. RandomNumberService.htm located in the RandomNumberService directory:

3) Open a web-browser and navigate to the *.htm page that you have copied into the virtual folder e.g.:

http://localhost/RandomGenerator/RandomNumberService.htm

4) If this file is displayed correctly then it indicates that everything is going well so far. The *.html page should display basic details about the Web Service you have just created. It will display the list of available methods (in this case it will display only HelloWorld as we haven’t manually added our custom method). More importantly, it will contain a link to the Web Services “Service Description” or WSDL document (actually it is a link to the RandomNumberService.dll which responds by sending the WSDL file that describes the Web Service – e.g. http://localhost/RandomNumberService/RandomNumberService.dll?Handler=GenRandomNumberServiceWSDL). Click on the ‘Service Description’ link. If the browser displays the WSDL file, everything is going very smoothly and you can move to “Step 3 – Creating a Web Service Client”. If this doesn’t happen, the most likely reason is that the virtual directory that you created has not been set up to handle dll’s. Try the following checklist (one at a time) using the properties Dialog of the virtual directory (right click on the virtual directory and select ‘Properties’:

a. In the Virtual Directory Tab, make sure that Execute Permissions is set to Scripts and Executables.

b. In the Directory Security Tab, click the Edit button and enable ‘Anonymous access’ and ‘Allow IIS to control password’.

c. In the Virtual Directory Tab click the ‘configure’ button which will bring up a new dialog. Navigate to the Mappings tab. Check to see if the extensions .dll and .srf are registered. If they aren’t (and they weren’t in one of our installations) then add them.

Step 3: Creating a Web Service Client

In this case we will create a web-based client using C# and Visual Studio. One of the advantages of using a Web Service is that almost any language and/or software can be used to call its methods and display the results. For example, they could be called from any *.exe program; from an applet or as is the case here from a ASP.NET web-page. We have chosen this option because it fits well with the goal of this project, namely to wrap up some C++ code (representing an existing model) into Web Service Methods so that the model can be run and the results made available across the www:

1) Open up a copy of Visual Studio (2005). Create a new web site project (File->New->Web Site. Select the ‘ASP .NET Web Site’ option from the resulting dialog box. Give the project a name (RandomClient in this case) and set the language to C#. Press OK to create the project. The project will be created with a single web-page - Default.aspx.

1) The next step is to create the code that will enable the web page to call the Random Web-Service methods. First save a copy of the WSDL file to the folder that contains the Web Service. Then from the Visual Studio menu select Website->Add Web Reference. In the URL textbox of the resulting dialog, type the URL of the WSDL (e.g. http://localhost/RandomNumberService/RandomNumberService.wsdl). Press GO and the methods described by the WSDL will be shown. Select a name for the Web Service reference (e.g. RandomService) and press OK. Two files will be added to your project under App_WebReferences: RandomNumberService.wsdl and RandomNumberService.discomap. These are the two files that Visual Studio uses to create the auto-generated code that makes it easy to call the Web Service methods. Now all we need to do is write a small amount of C# code to call the methods.

2) Associated with the Default.aspx page is a Default.aspx.cs file that contains the code for the page. Navigate to this file. Just below all the ‘using’ statements add the following:

using System;using System.Data;
using System.Configuration;
using System.Web;
using System.Web.Security;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Web.UI.HtmlControls;
//Add this line of code:
using RandomService;

where RandomService is the name you gave to the Web Service reference from step 2. Then, in the class declaration, add the following code:

public partial class _Default: System.Web.UI.Page
{ 
    String formattedstring;
    protected void Page_Load(object sender, EventArgs e)
    {
        RandomService.RandomNumberServiceService theservice;
        theservice = new RandomService.RandomNumberServiceService(); 
        formattedstring    = theservice.GenRandomNumbers (0,10, 10);
        //do any amount of formatting on the string so that it
        //can be displayed nicely on a web-page e.g.
        //formattedstring.Replace("\n","<BR>");
    }
}

This code adds a member variable (of type string) to the page called formattedstring. In the Page_load event handler, an instance of the auto-generated Web Service class is created which is used to call the method GenRandomNumbers(…) and the results are stored in the String formattedstring.

3) Finally, display the string of random numbers in the Web Page. To achieve this, navigate to the Default.aspx page and modify the auto-generated code so it looks as follows:

 <%@ Page Language="C#" AutoEventWireup="true" CodeFile="Default.aspx.cs" Inherits="_Default"%>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml"> 
<head runat="server">
    <title>Untitled Page</title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
    <strong> 
    Random Numbers
Produced by Web Service:<br />
    </strong>
    <%=formattedstring%>
    <br />
    </div>
    </form>
</body>
</html> 

4) Finally, save all the files and navigate to the Default.aspx web page using a web-browser. The page should look something like:

Ten random numbers have been generated between 0 and 10 (as per the call to the Web-Service method hard-coded in the code file). If the page is refreshed, a new set of numbers will be displayed (following a new call to the Service). In this case, the numbers (output from our imaginary model) are displayed very plainly, but it is easy to see how the string could be formatted either within the ASP.NET code or at the Web Service side to produce a better looking result. Similarly, it is easy to see how the call to the web-service method could be made after gathering some input from a web-form.

At this point, the basic goals of the project have been achieved. The article has demonstrated how to:

a) Wrap up some C++ code (representing a complex model that we don’t want to recode) within an ATL Web Service.

b) Implemented a web-page based client that calls this model remotely across the internet, receives some output data and displays it on a web-page.

Given this basic framework, the project could be expanded in a number of ways. First and foremost, the simple, trivial example of creating random numbers could be replaced with much more important and useful models. The client be developed into a pleasant user interface that collects user input, calls the Web Service and displays the results in a nice GUI. Finally, the Web Service could send different types of data to the client. For example, in the sample project we have included a method that uses some C++ code to create a simple plot of some random numbers (again pretending that this is the output from a much more complicated C++ coded ecological model) and included a method in the web service that sends the bitmap bytes (as a byte array) to a web-based client. Once the byte array has been delivered to the client, it simply needs to reassemble it as a bitmap and display it on the web page (again, just a few lines of code). Using this method video, GIS, Excel files containing data or any other information could be sent by the Web Service. This has the potential to greatly enhance the interpretation and dissemination of model results. The figure below shows a more complex web page with formatting for the text string and display of the bitmap.

Points of Interest

Documentation for Web Services (especially ATL) is poor. We hope that, at the very least, our article provides the motivation for other researchers to look into Web-services and chase down some of this documentation. The poor documentation for ATL is probably a result of the development of newer technologies (.NET web services etc). We used ATL because we didn’t want to have to mix managed and unmanaged code (we wanted our legacy code to compile as is). The important thing to remember is that, however useful we found these methods, there are probably many alternatives available. See other code project articles for examples.

In particular we found an annoying niggle in the interaction between the ATL web-service and modern WSDL parsers (e.g. Microsoft Visual Studio’s, Eclipse). The default code generated by the ATL wizard in Visual Studio creates a WSDL file that is not compatible with the W3C SOAP standard. By default the WSDL file is created as rpc/encoded style. One tiny change in code is all that is needed to change the style to document/literal and get the WSDL into a format suitable for modern WSDL parsers (see step 6 of Creating the Web Service). One tiny bit of code….three days of going backwards and forwards through obscure SOAP, Visual Studio, Eclipse and any other documentation we could find. If all that this article achieves is to provide a warning and a fix for this problem will be very happy.

IIS is a bit odd. On our development machine the *.dll ‘s of the Web Service server worked first time. Then we switched to another machine, set up IIS, installed the *.dll’s and nothing worked. Eventually, we tracked down the problem to IIS not having the *.dll and *.srf extensions properly registered. We are still unsure why IIS was set up differently on one computer than another, but the problem was made more difficult to resolve because the tool in IIS for registering these extensions has a fault that makes it look like it is impossible to add mappings (OK button disabled). See the online documentation here:

http://support.microsoft.com/kb/317948

Overall, one puzzling question we had during almost every step of this project is why do WSDL documents have to be so complicated? The concept of SOAP, Web Services and the WSDL seems so simple. The Web service server takes a ‘package’ of information from a client and returns information back to the client. This information is in a consistent, standardized data format so that communication is possible irrespective of languages and platform difficulties. So the WSDL file represents an unambiguous description of where the service is located, the inputs it takes and the outputs it returns (including data types) so that anybody can implement their own client. Undoubtedly a brilliant idea. But why does the WSDL have to be so complicated? Wouldn’t it be good if the WSDL could just as easily be written by hand (to overcome problems with parsers) and if it provided an overview of the service even at first glance? All we see in a WSDL document is an unreadable jumble of tags and acronyms.

Overall, we conclude that Web Services fulfill the requirements we set ourselves at the beginning of this article:<o:p>

1) They have allowed us to wrap up old C++ code into code that allows the results of the models to be distributed across the web. This process, given the wizards provided by Visual Studio, is relatively easy although we are still likely to encounter some problems (mixing ATL with MFC code for example).

2) They allow web-based clients to be built very easily. In addition, once the methods exposed by the Web Services are known, these clients can be built by any programmer with VB, Java or C#. Different clients can be built (perhaps for different customers) that all call the same Web Services but that obtain input and display the results slightly differently.

<o:p> We have successfully used this framework to create Web Services and clients that implement Stochastic Weather Generator models (Models that provide unique weather data for a region) and complex population models that predict the occurrence, abundance and damage caused by pest species.

<o:p>The major advantage to an ecologist is that we can now develop models in our ‘Native’ languages (the ones we are most comfortable with), look after revisions easily (since we are dealing with only one server side version of the model versus lots of distributed *.exe files), and easily distribute these models over the internet.

History

none yet.