WorkflowChannel for Windows Azure BizTalk Service (WABS)

Roman Kiss

5.00/5 (7 votes)

Jan 25, 2014

CPOL

22 min read

33263

155

This article describes how a Windows Azure BizTalk Service Bridge pipeline can be extended for message mediation by Workflow (VETER + WORKFLOW Pattern).

Download source

Features
Introduction
Concept and Design
Implementation
Usage and Test
Conclusion
References

Features

WABS Bridge Hosting
VETER + WORKFLOW = VETEW Pattern
Loosely decouple pattern
Based on WCF Custom Channel
Encapsulation of the Workflow Definitions
Workflow Definitions (xaml) stored in the Azure Blob
Using a blob metadata for workflow configuration
Message Exchange Pattern via workflow arguments
Full transparent pattern to the VETER
Allow to use Workflow Custom Activities

Microsoft recently released the general availability (GA) of Windows Azure BizTalk Services (WABS) in the Windows Azure Platform. WABS enables business integrations using a declarative programming within the pre-built message pipeline pattern known as VETER (Validate-Enlarge-Transform-Enlarge-Route) in the multitenant cloud environment. Received message by source entity flows via VETER pattern to the destination entity such as Windows Azure Service Bus Relay Endpoints. Service Bus Queue/Topic, Azure Blobs, FTP/SFTP Endpoints and External Endpoints via WCF Stack.

WABS represents a bridge between the source and destination endpoints, where an inbound message can be targeted only at one of the many destination endpoints. The message pipeline is initiated in the memory and all states are persisted only in memory and/or message, therefore the WABS Bridge is a stateless scenario and the message lifetime process is short as much as possible. In other words, WABS Bridge has a responsibility to mediate and deliver a message to the destination in a very short time within the message lifetime. In the case of the bridge crash, error, etc. the error message is returned back to the caller with a detail reason.

For this kind of pattern, the WABS Bridge is unified and simplified using a VETER pattern. This simplicity allows to hide the bridge complexity and technology integration stack while focusing more on the business scenario such as message validation and mediation. The WABS built-in VETER pattern enables the WABS Bridge to use in the Pre-Processing and Post-Processing business integration stack, where the centralized processing part is holding a state for business composition. Any technology can be used for the business centralized part, such as BizTalk Server, IBM WebSphere ESB, Oracle ESB, etc. or custom built composite application.

Basically, the WABS is a cloud web role service with pre-built boilerplate stack for Enterprise Application Integration (EAI) scenario to run in a secure, auto-scalable, dedicated and per-tenant environment. Besides that, the bridge boilerplate has built-in monitoring and tracking capabilities. The VETER bridge pattern has pre-built extensibility points such as On Enter Inspector and On Exit Inspector, where the state can be extended by a custom code. This extension is imperative and must be considered during the design time, in other words it must be embedded into the bridge.

On the other side, the WABS Bridge Pattern allows to extend a message pipeline using a loosely decouple manner, fully transparent and declarative way such as using a WCF ExternalServiceEndpoint. This is a destination (client) proxy WCF channel for outbound message. Using this channel in the bridge pattern, the outbound message is forwarded to the target endpoint, which can be considered as a remote mediator, business processor, etc. The ExternalServiceEndpointis based on the WCF stack, therefore we have an additional extensibility point within the WABS Bridge Pattern.

Notice, that the ExternalServiceEndpoint is for outbound message only. In other words, this channel is only for the client (proxy), so there is no need to have any listener in the channel stack. From the hosting point of the view, the WABS represents a hosting process for Destination proxies. This is a great feature of the Windows Azure BizTalk Services.

Now, coming to the conceptual point of the extensibility WABS Bridge Pattern for Workflow, where the EnternalServiceEndpoint allows to plug-in a WCF Custom Channel for outbound message.

Having a special WCF Custom Channel for Workflow invoker, the outbound message can be forwarded to the Workflow skeleton, declared by xaml definition and executed within the WABS hosting process.

This article describes this process. It shows you how easily WABS Bridge can be extended for this feature in the message pipeline such as Workflow, where the message can be mediated with a workflow orchestration capability in the full transparent and loosely decupled manner. Using a WCF custom WorkflowChannel in the WABS Bridge is giving you a new pattern such as VETER + WORKFLOW = VETEW pattern.

I do strongly recommend reading my previous articles such as WorkflowChannel for Routing Service (detail WorkflowChannel description), RoutingService on Azure and Azure Virtual Bridge. The last article is conceptually very close to the WABS Bridge, where WCF Routing Service is hosted in the Azure Web Role.

Within the Visual Studio Bridge Designer, there is no change and as you will see later, only config file for a specific destination needs to be configured for using a custom WorkflowChannel.

The following picture shows an example of the message Itinerary for 3 different bridges, where an outbound message is routed to the wcf custom WorkflowChannel.

The above picture (message Itinerary) shows that the outbound message is sent to the External Service Endpoint, so the message should flow to the external service, where it will be executed outside of the WABS runtime domain.

But, wait a moment. Until we don't see each destination config file for a specific External Service Endpoint, it doesn't have to be true, because of the plug-in the WCF custom WorkflowChannel. The outbound message will not exit the WABS runtime domain, thus the outbound message will be executed in the WABS hosting process, but under the Workflow orchestration, not under the VETER pattern.

This is the conceptual extension and from the abstraction point of the view, I can call this pattern VETEW, where R-outer is replaced by W-orkflow. Note, that the W represents an in-memory workflow, its lifetime is done by WABS lifetime. In other words, this is a non-persisted Workflow run in the WABS Bridge.

That's great. Let's continue. The following picture shows a simple example of the Pass-Through Bridge to the WorkflowChannel, where the outbound message is mediated for Publisher, for example writing a message payload to the Azure Blob and notification on the Service Bus Topic:

The above picture shows additional comments that indicate the destination connectivity is calling a custom channel (net.wf://repository/bridges) on One-Way External Service Endpoint. So, any inbound message received by this bridge (Fire) is forwarded to the Custom WorkflowChannel (Publisher). Basically, the above picture shows a basic pattern without the full VETER message mediation, it is a pass-trough workflow bridge.

The next picture shows more capable message mediation within the WABS Xml Request-Reply Bridge. The inbound message can be fully mediated in the Router entity such as Xml Request-Replay Bridge and then its outbound message is forwarded to the Custom WorkflowChannel.

In the above message itinerary, the destination entity is called as WorkflowInvoker. The reason for that is, that any inbound message received by Router can invoke a specific xaml workflow definition, as you will see latter.

Now, it is a time to show a custom config file for WorkflowChannel:

As you can see, the configuration of the client endpoint with a custom binding is very straightforward. The client address net.wf:://repository/bridgesdeclares a custom binding with net.wf schema and a xaml resource stored in the repository (Azure Blob) below the container bridges. Authorization access to the repository account is declared in the binding section.

Really, this is very straightforward in the 1-2-3 steps such as adding a custom WorkflowChannel assembly to the bindingElementExtension, then adding a customBinding and lastly configuring a client endpoint for WorkflowChannel. Basically, this configuration can be used for any WorkflowChannels in your Bridges where all xaml workflow definitions are stored in the same place.

Behind the WorkflowChannel is a xaml workflow definition stored in the Repository, The WorkflowChannel has a responsibility to load and execute a specific xaml definition from the storage.

The following picture shows an example of the very simple workflow such as a message LoopBack, where an incoming message is passed back to the invoker.

The LoopBack Workflow is stored in the container bridgesand has only one Assign activity for message loopback. Note, that the _inMessage and _outMessage are Workflow mandatory Arguments used for message integration with a Bridge, as you can see later in more detail. Note, that this LoopBack Workflow is very useful piece for creating a custom message mediator primitive within the VETER pattern, where a transformation is a place to provide a complex message mediation to encapsulate a business model. In additional, the LoopBack Workflow can participate in the business mediation as well.

Let's bring up the following picture to show a position between the VETER Bridge and Workflow Definitions:

The above picture is a great conceptual picture of the VETEW Pattern. The VETER Bridge part represents a Pre/Post Message Processing for Workflow Definition. The xaml workflow resources, known as the workflow bridge definitions are stored in the Azure Blob Storage (repository). We can deploy the WABS Bridge once for a business oriented library of the workflow definitions and call them to compose a business model based on the some rule sequence.

Of course, the WABS Bridge can be built for more complexity, using a filtering, for example, like it is shown in the following picture:

The above Bridge shows a routing message to the 2 custom destinations such as WorkflowChannels. Based on the Route Action the message can be routed to the Weather or LoopBack.workflow definitions.

Wait a moment; how does the custom WorkflowChannel know which workflow definition to use?

Well, as you may know from my previous articles, the name of the xaml workflow definition has been passed via an address uri query string, for instance:

Unfortunately, the current version of the WABS doesn't allow to pass a query string via an address Uri, therefore the original WorkflowChannel for Routing Service was necessary to modify its implementation for accepting a custom header in the message in the case if the address Uri doesn't have a query string. I hope, the Microsoft WABS Team will allow this feature in the next version.

Anyway, for now, the message forwarding to the WorkflowChannel must have a simple string type custom header query, see the following screen snippet of the Bridge Message Sender program:

The above example shows, how any workflow definition can be invoked behind the Router Bridge. The Request message has a direct custom header query for invoking a xaml workflow, in this example it is a LoopBack workflow. As you will see later, the query header can also specify a name of the container, etc.

That's great, what about a runtime invoking a workflow definition from the repository based on the message payload, etc.

Basically, we can use the same principle for adding and modifying a header in the message. The following example shows how a custom header can be added into the outbound message:

In the above example, the header value is created explicitly in the Expression TextBox. Note, that the value is overwritten for existing header.

I hope by this point you get an idea of what Wmeans in the Bridge VETEW Pattern.

As I mentioned earlier, the WABS Bridge is a stateless bridge to make a Pre and/or Post Message Processing for mediation primitives, transformations, enlarge, etc. Adding the W into the VETER Pattern we can enable WABS Bridges for decomposition of the business model which will require a message splitter, message accelerator, business context streaming, etc. without using more sophisticated (and expensive business server) for small and/or middle size solutions.

The following picture shows an example from my previous article Using Azure Lease Blob, where the business composition is done using an EventStream based on the Lease Blob.

As you can see, the above picture shows a composition of the Business Model based on the EventStream, where each WABS Bridge exclusively published an Eventto the EventStream, therefore each Publisher can see a business context state for message acceleration, triggering events, etc.

In the above introduction section I mentioned few my previous articles. Allow me to add one more article (WF4 Custom activities for message mediation) that will help you with the message mediation in the Workflow. This article will help you to build custom message mediators, invoker based on the Contract-First, Splitter, Analyzers, etc. in the Sequence and/or Flowchart fashion.

OK, let's continue with the concept and design. I am assuming you have a working knowledge of the Windows Azure Platform (BizTalk Services, Azure Blob, Service Bus, etc.)

Concept and Design

The concept of adding a Workflow into the WABS Bridge is based on the outbound WCF Custom Channel used in the message itinerary for generic destination WCF External Service Endpoint. Plug-in the WCF Custom Workflow Channel to the destination proxy enables the message exchange pattern - MEP (one-way or two-way) between the VETER and Workflow Pattern in sync or async manner.

The following picture shows an example of the simple one-way message pattern:

The outbound message from the VETER is forwarded to the One-Way External Service Endpoint configured for custom WorkflowChannel, see the following configuration:

The message flows into the Workflow orchestration. As you can see in the above message itinerary, the message in the Workflow can be processed in the custom "second" VETER pattern orchestrated by workflow activities in the Sequence and/or Flowchart skeleton.

The WorkflowChannel has a responsibility to load and invoke a xaml workflow definition based on the query string located in the address Uri or message header. The source repository can be configured for default place such as a container in the Azure Blob (this article support only this feature) or folder in the project, for instance: Artifacts.

The query string contains at least one name-value pair such as the name of the xaml workflow definition. In addition, this is the place to configure loading of the xaml workflow definition, specifically by the request. The following are few examples of the query string:

xaml=LoopBack
xaml=LoopBack&amp;repository=bridges
xaml=LoopBack&amp;repository=bridges&settings=bridges/Settings/MySettings 
xaml=LoopBack&amp;repository=bridges&amp;ns=mynamespace&amp;
     st=2014-01-18T20%3A00%3A51Z&amp;se=2014-01-18T21%3A00%3A51Z&amp;sr=c&amp;sp=r&amp;sig=yFriLn2xiJ...

Note, that the last example of the query string shows an example of the blob addressing with a SAS arguments.

Storing a xaml workflow definition in the Azure Blob is the default and the preferred way. The major reason for that is the blob metadata, where the workflow configuration can be stored. The following picture shows a data integration between the VETER and Workflow:

As you can see, the outbound VETER message is flowing to the Workflow via an _inMessage and returned back via an _outMessage arguments. There is a third mandatory workflow argument such as _metadata. The WorkflowChannel populated the input arguments such as _inMessageand _metadata arguments and based on the MEP fashion, it is using an _outMessageargument for Reply message. Also, if the xaml workflow resource is an Azure Blob resource, then all blob's metadata are copied to the _metadata argument. Using the blob metadata for workflow configuration is a great feature, it enables us to decompose the workflow variables in the business model and share them across the multiple workflows knows as Enterprise Variables. This is an advanced feature of the custom WorkflowChannel, see more details about this scenario in my article Azure Virtual Bridge.

This article includes a full source code of the WCF Custom WorkflowChannel and it is very easy to modify loading of the xaml resource based on the requirements, for instance, a xaml definition from specific folder/database/blob/etc. and its metadata from other sharable (centralized) database/blob/folder/etc.

Custom message header - query

The custom message header queryis a string type qualified header for declaring a xaml workflow definition such as name, location, etc. Every outbound message forwarded to the WorkflowChannel Endpoint must have this header in the message.

The following snippet shows an example of the qualified header query:

 <query xmlns="urn:rkiss/2014/wf">xaml=Weather</query>

The header can also be created declaratively during a runtime in the VETER Pattern or Route Action.

The following screen snippet shows an example of the routing a message to the Weather workflow definition in the case of zip > 90000:

Implementation

The WCF custom channel for Workflow has a very simple model supporting only IOutputChanneland IRequestChannel interfaces:

All custom logic related to the workflow target is encapsulated into the custom WfHelper static class, see the following code snippet:

public IAsyncResult BeginRequest(Message message, TimeSpan timeout, AsyncCallback callback, object state)
{
  base.ThrowIfDisposedOrNotOpen();
  base.RemoteAddress.ApplyTo(message);

  // load and create instance of workflow
  WorkflowApplication wfa = 
        WfHelper.CreateWorkflowApplication(message, repositorySettingName, trackingParticipent);

  // run workflow asynchronously
  return wfa.BeginRun(timeout, callback, state);
}   
public Message EndRequest(IAsyncResult result)
{
  return WfHelper.EndRun(result);
}

At this point, I would like to refer you to my previous articles where detail description of the design and implementation of the WorkflowChannel is described. There are few changes of the original WorkflowChannel made for WABS Bridge are couple, such as signing an assembly and adding a query header logic in the GetQuery Method:

// query from header
if (string.IsNullOrEmpty(query))
{
    int index = message.Headers.FindHeader("query", "urn:rkiss/2014/wf");
    if (index >= 0)
        query = message.Headers.GetHeader<string>(index);
}

Let's spend more time with Usage and Test of the WorfklowChannelin the WABS Bridge.

Usage and Test

First of all, the following are prerequisites (based on my development environment):

Visual Studio 2012 Update 4
Install Windows Azure SDK for .NET 2.2 (Visual Studio 2012)
Windows Azure Platform account
WABS namespace
Http/s connectivity to the Internet
Downloading package for this article
Azure Storage Explorer
Install Windows Azure BizTalk Service Explorer
Knowledge and experience with Windows Azure Platform (Blobs, WABS, Service Bus, etc.)

In this section, I am going to show you step by step usage of the WorkflowChannel in the WABS Bridge.

Step 1. Download a package from this article

The following screen snippet shows BizTalkServiceVirtualBridge solution package included in this article:

As you can see, the above solution includes 2 projects such as a WABS project - BizTalkServiceVirtualBridgeand WokflowChannel library project. Basically for usage of the WorkflowChannel in your next WABS solution only the WorkflowChannel.dll assembly is required. Therefore, the above WABS package is only for our test purposes of the WCF Custom WorkflowChannel. As you can see, there is a folder Tools with some assemblies used for our testing. One more thing, the Artifacts folder has two xamlx resources for our testing such as LoopBack and Weather. These workflows are for testing purposes only.

Do not compile this solution before the step 2, You have to update the solution with your accounts.

Step 2. Modify solution for your accounts.

The solution must be modified for your accounts such as BizTalk Service URL property and configuration file for all following destination endpoints:

As I mentioned earlier, the WorkflowChannel is using an Azure Blob Storage for repository of the xaml workflow definitions. For test purposes, this solution uses (like it is shown in the above config) two containers, one is for assemblies and the other one is bridges for xaml resources. Please, create those private containers in your blob storage.

Once these changes are made, the solution is ready for deployment to your WABS endpoint.

One more help, if you will have an issue with building this solution, please close the Visual Studio and reopen it again, but use only the Build feature. I believe there is a bug in the Visual Studio 2012 when holding assembly reference. That's my experience and I hope that upcoming WABS for Visual Studio 2013 will resolve it.

Step 3. Deployment

To deploy WABS solution from the VS2012, the following dialog prompt will be show up on your screen:

The deployment Endpoint will show your WABS namespace and you must fill in the AcsNamespace and SharedSecret properties. After that, you can watch the deployment process in the Ouput panel and I hope without any errors.

If the deployment succeeded, you should see the following in the Server Explorer this solution in your WABS namespace:

The Windows Azure BizTalk Service Explorer is a great tool. It allows to refresh, restart, update, etc. your WABS. Also, you can see a list of the uploaded assemblies. The above picture shows those assemblies which we need to have for our next testing. Note, that the missing assemblies can be manually uploaded or added in the solution reference with the flag True for Copy Local property. By the way, this flag is useful for controlling an assembly upload, it will speed up the re-deployment process when the bridge itinerary has changed, for instance.

Now we can focus on Workflow part.

Before we proceed, let's pre-test for our deployed WABS which will indicate if we are on the right path to the Workflow.

Let's invoke some Workflow, at this moment we can invoke any kind of name because our blob storage is empty.

Step 4. Invoking a non-existing Workflow

For easy way to send a message to the WABS, we can use my "dirty tool" included in the solution folder Tool.

Please, launch the MessageSender program and populate your account values for your WABS namespace. Setup a query string value for xaml=ABCD. After that, press the button Send. You should get a response message from your WABS indicated the following:

That's a good sign, that your WABS should work in the following steps.

Step 5. Invoking an existing Workflow

In this step we are going to upload blobs to your Azure Storage. Note, you will need to use your own credit for WABS and Blob Storage.

Any 3rd party Azure Storage Explorer, but in this step let's used very popular and free Azure Storage Explorer from my friend David Pallmann.

Please, upload the following xaml resources to your Blob Storage container bridges:

Artifacts/LoopBack.xamlx
Artifacts/Weather.xamlx

make the following changes:

delete an extension .xamlx in the name of the blob (this is an option, but in this case we have to use a full name of the xaml resource in the query string)
use content type application/xaml+xm (this is an option in the opposite type text/plain, where we can redirect a resource to another blob)
add the following metadata for Weather blob:

Please, upload the following assemblies to your Blob Storage container assemblies:

Tools/MessageMediationActivityLibrary.dll

make the following changes:

delete an extension .dll in the name of the blob

In the case of using a custom activities in your Workflow, you have to upload its assembly into this container and restarted a WABS be sure that this assembly is going to load into the WABS Bridge process (web role).

That's all. Now it is the time to do some testing. Let's start with something very simple and using only a Microsoft Workflow Activities from the .Net 4.5, some simple workflow such as a LoopBack.

Step 6. Test LoopBack Workflow (Router)

This test is using a Router WABS Bridge. Its address ishttps://your_namespace.biztalk.windows.net/default/Router

Any query string will be forwarded into the WorkflowInvoker destination with a WCF custom WorkflowChannel.

Use the MessageSender program to send a request message with a query header:

<query xmlns="urn:rkiss/2014/wf">xaml=LoopBack</query>

The following screen snippet shows a response from the LoopBack WABS Bridge:

You can work with different zip code in the request message payload to see the changes in the response message. The response panel shows a TrackingID. This id can be used for tracking a message through the Router Bridge using the WABS Explorer or Azure Portal.

Note, that this Router Bridge can be used for invoking any xaml workflow definition stored in the container bridges. I am sure, you will use it to invoke your workflows as well.

Step 6. Test Weather Workflow (WeatherBridge)

This test will demonstrate the capability of the VETER bridge to select two different xaml workflow definitions from the Azure Blob Storage. The Weather Workflow is a very straightforward workflow using a custom activity CreateMessage(from my recommended article WF4 Custom activities for message mediation) to create a request and response message from/to call to the Weather public service. In this test we are testing the capability of the loading a custom assembly from the container assemblies.

The following picture shows a WeatherBridge with two destinations configured for WorkflowChannel:

Note, that the inbound message doesn't need to have a query header, because this header is added in the outbound message based on the filter.

As you can see, based on the payload value ZIP, the message is routing to the properly destination. There are 3 groups of the filters. The third one is when ZIP = 90000. In this case, the caller will receive an error message.

The following screen snippets show these cases:

LoopBack Workflow:

Weather Workflow:

As you can see, there is a custom header in the last response message configured in the weather's metadata. You can use this configuration during your testing without re-deploying the bridge or updating blob content.

That's all for testing. I hope both tests succeeded on your WABS deployment. In addition, the demonstration showed you the capability of the VETEW Pattern using the WorkflowChannel.

Advanced Tooling

My article RoutingService on Azure is described a small modification of the Azure Storage Explorer which will associate a program with blob content. The following picture shows this user interface extension:

Clicking on the Edit link, the re-hosted WorkflowDesignerTester will launch for editing of the xaml definition. The changes in the xaml can be saved in the blob. Note, that the fresh version of the WorkflowDesignerTester program can be found in the solution Tools folder.

Conclusion

This article described how we easily we can extend the pre-built VETER Pattern in the Windows Azure BizTalk Service (WABS) by Workflow (WF) capability. Using a WCF Custom WorkflowChannel in the Destination Endpoint, the outbound VETER message pipeline continues in the Sequence and/or Flowchart skeleton of the Workflow. This article described a new concept pattern such as VETER + WORKFLOW = VETEW. I hope you enjoyed it.