WorkflowChannel for Windows Azure BizTalk Service (WABS)





5.00/5 (7 votes)
This article describes how a Windows Azure BizTalk Service Bridge pipeline can be extended for message mediation by Workflow (VETER + WORKFLOW Pattern).
Contents
Features
- WABS Bridge Hosting
- VETER + WORKFLOW = VETEW Pattern
- Loosely decouple pattern
- Based on WCF Custom Channel
- Encapsulation of the Workflow Definitions
- Workflow Definitions (xaml) stored in the Azure Blob
- Using a blob metadata for workflow configuration
- Message Exchange Pattern via workflow arguments
- Full transparent pattern to the VETER
- Allow to use Workflow Custom Activities
Introduction
Microsoft recently released the general availability (GA) of Windows Azure BizTalk Services (WABS) in the Windows Azure Platform. WABS enables business integrations using a declarative programming within the pre-built message pipeline pattern known as VETER (Validate-Enlarge-Transform-Enlarge-Route) in the multitenant cloud environment. Received message by source entity flows via VETER pattern to the destination entity such as Windows Azure Service Bus Relay Endpoints. Service Bus Queue/Topic, Azure Blobs, FTP/SFTP Endpoints and External Endpoints via WCF Stack.
WABS represents a bridge between the source and destination endpoints, where an inbound message can be targeted only at one of the many destination endpoints. The message pipeline is initiated in the memory and all states are persisted only in memory and/or message, therefore the WABS Bridge is a stateless scenario and the message lifetime process is short as much as possible. In other words, WABS Bridge has a responsibility to mediate and deliver a message to the destination in a very short time within the message lifetime. In the case of the bridge crash, error, etc. the error message is returned back to the caller with a detail reason.
For this kind of pattern, the WABS Bridge is unified and simplified using a VETER pattern. This simplicity allows to hide the bridge complexity and technology integration stack while focusing more on the business scenario such as message validation and mediation. The WABS built-in VETER pattern enables the WABS Bridge to use in the Pre-Processing and Post-Processing business integration stack, where the centralized processing part is holding a state for business composition. Any technology can be used for the business centralized part, such as BizTalk Server, IBM WebSphere ESB, Oracle ESB, etc. or custom built composite application.
Basically, the WABS is a cloud web role service with pre-built boilerplate
stack for Enterprise Application Integration (EAI) scenario to run in a secure,
auto-scalable, dedicated and per-tenant environment. Besides that, the bridge
boilerplate has built-in monitoring and tracking capabilities. The VETER
bridge pattern has pre-built extensibility points such as On Enter
Inspector
and On Exit Inspector
, where the state can be
extended by a custom code. This extension is imperative and must be considered
during the design time, in other words it must be embedded into the bridge.
On the other side, the WABS Bridge Pattern allows to extend a message
pipeline using a loosely decouple manner, fully transparent and declarative
way such as using a WCF ExternalServiceEndpoint
. This is a destination
(client) proxy WCF channel for outbound message. Using this channel in the
bridge pattern, the outbound message is forwarded to the target endpoint, which
can be considered as a remote mediator, business processor, etc. The
ExternalServiceEndpoint
is based on the WCF stack, therefore we have an
additional extensibility point within the WABS Bridge Pattern.
Notice, that the ExternalServiceEndpoint
is for outbound
message only. In other words, this channel is only for the client (proxy), so there is no need
to have any listener in the channel stack. From the hosting point of the view,
the WABS represents a hosting process for Destination
proxies. This is a
great feature of the Windows Azure BizTalk Services.
Now, coming to the conceptual point of the extensibility WABS Bridge Pattern
for Workflow, where the EnternalServiceEndpoint
allows to
plug-in a WCF Custom Channel for outbound message.
Having a special WCF Custom Channel for Workflow invoker, the outbound message can be forwarded to the Workflow skeleton, declared by xaml definition and executed within the WABS hosting process.
This article describes this process. It shows you how easily WABS Bridge
can be extended for this feature in the message pipeline such as Workflow, where the
message can be mediated with a workflow orchestration capability in the full
transparent and loosely decupled manner. Using a WCF custom WorkflowChannel in
the WABS Bridge is giving you a new pattern such as
pattern. VETER + WORKFLOW
= VETEW
I do strongly recommend reading my previous articles such as WorkflowChannel for Routing Service (detail WorkflowChannel description), RoutingService on Azure and Azure Virtual Bridge. The last article is conceptually very close to the WABS Bridge, where WCF Routing Service is hosted in the Azure Web Role.
Within the Visual Studio Bridge Designer, there is no change and as you will see later, only config file for a specific destination needs to be configured for using a custom WorkflowChannel.
The following picture shows an example of the message Itinerary for 3 different bridges, where an outbound message is routed to the wcf custom WorkflowChannel.
The above picture (message Itinerary) shows that the outbound message is sent to the External Service Endpoint, so the message should flow to the external service, where it will be executed outside of the WABS runtime domain.
But, wait a moment. Until we don't see each destination config file for a specific External Service Endpoint, it doesn't have to be true, because of the plug-in the WCF custom WorkflowChannel. The outbound message will not exit the WABS runtime domain, thus the outbound message will be executed in the WABS hosting process, but under the Workflow orchestration, not under the VETER pattern.
This is the conceptual extension and from the abstraction point of the view, I
can call this pattern VETEW
, where R-outer is replaced by
W-orkflow. Note, that the W represents an in-memory workflow, its lifetime is
done by WABS lifetime. In other words, this is a non-persisted Workflow run in
the WABS Bridge.
That's great. Let's continue. The following picture shows a simple example of the Pass-Through Bridge to the WorkflowChannel, where the outbound message is mediated for Publisher, for example writing a message payload to the Azure Blob and notification on the Service Bus Topic:
The above picture shows additional comments that indicate the
destination connectivity is calling a custom channel (net.wf://repository/bridges
)
on One-Way External Service Endpoint. So, any inbound message received by this
bridge (Fire
) is forwarded to the Custom WorkflowChannel (Publisher
).
Basically, the above picture shows a basic pattern without the full VETER
message mediation, it is a pass-trough workflow bridge.
The next picture shows more capable message mediation within the WABS
Xml Request-Reply Bridge
. The inbound message can be fully mediated in the
Router entity such as Xml Request-Replay Bridge
and then its
outbound message is forwarded to the Custom WorkflowChannel.
In the above message itinerary, the destination entity is called as
WorkflowInvoker
. The reason for that is, that any inbound message
received by Router
can invoke a specific xaml workflow definition,
as you will see latter.
Now, it is a time to show a custom config file for WorkflowChannel:
As you can see, the configuration of the client endpoint with a custom binding
is very straightforward. The client address net.wf:://repository/bridges
declares a custom binding with net.wf
schema and a xaml
resource stored in the repository (Azure Blob) below the container bridges
.
Authorization access to the repository account is declared in the binding
section.
Really, this is very straightforward in the 1-2-3 steps such as adding a custom
WorkflowChannel
assembly to the bindingElementExtension
,
then adding a customBinding
and lastly configuring a client
endpoint for WorkflowChannel. Basically, this configuration can be used for any
WorkflowChannels
in your Bridges where all xaml workflow
definitions are stored in the same place.
Behind the WorkflowChannel is a xaml workflow definition stored in the Repository, The WorkflowChannel has a responsibility to load and execute a specific xaml definition from the storage.
The following picture shows an example of the very simple workflow such as a
message LoopBack
, where an incoming message is passed back to the
invoker.
The LoopBack
Workflow is stored in the container bridges
and has only one Assign activity for message loopback. Note, that the
_inMessage
and _outMessage
are Workflow mandatory
Arguments used for message integration with a Bridge, as you can see later in more
detail. Note, that this LoopBack Workflow is very useful piece for creating a
custom message mediator primitive within the VETER pattern, where a
transformation is a place to provide a complex message mediation to encapsulate
a business model. In additional, the LoopBack Workflow can participate in the business mediation
as well.
Let's bring up the following picture to show a position between the VETER Bridge and Workflow Definitions:
The above picture is a great conceptual picture of the VETEW
Pattern. The VETER Bridge part represents a Pre/Post Message Processing for
Workflow Definition. The xaml workflow resources, known as the workflow bridge
definitions are stored in the Azure Blob Storage (repository). We can deploy the
WABS Bridge once for a business oriented library of the workflow definitions and call
them to compose a business model based on the some rule sequence.
Of course, the WABS Bridge can be built for more complexity, using a filtering, for example, like it is shown in the following picture:
The above Bridge shows a routing message to the 2 custom destinations such as
WorkflowChannels.
Based on the Route Action the message can
be routed to the Weather
or LoopBack
.workflow
definitions.
Wait a moment; how does the custom WorkflowChannel know which workflow definition to use?
Well, as you may know from my previous articles, the name of the xaml workflow definition has been passed via an address uri query string, for instance:
Unfortunately, the current version of the WABS doesn't allow to pass a query string via an address Uri, therefore the original WorkflowChannel for Routing Service was necessary to modify its implementation for accepting a custom header in the message in the case if the address Uri doesn't have a query string. I hope, the Microsoft WABS Team will allow this feature in the next version.
Anyway, for now, the message forwarding to the WorkflowChannel must have a simple string type custom header query, see the following screen snippet of the Bridge Message Sender program:
The above example shows, how any workflow definition can be invoked behind
the Router Bridge. The Request message has a direct custom header query for
invoking a xaml workflow, in this example it is a LoopBack
workflow. As you will see later, the query header can also specify a name of
the container, etc.
That's great, what about a runtime invoking a workflow definition from the repository based on the message payload, etc.
Basically, we can use the same principle for adding and modifying a header in the message. The following example shows how a custom header can be added into the outbound message:
In the above example, the header value is created explicitly in the Expression TextBox. Note, that the value is overwritten for existing header.
I hope by this point you get an idea of what W
means in the Bridge VETEW
Pattern.
As I mentioned earlier, the WABS Bridge is a stateless bridge to make a Pre
and/or Post Message Processing for mediation primitives, transformations,
enlarge, etc. Adding the W
into the VETER
Pattern we
can enable WABS Bridges for decomposition of the business model which will
require a message splitter, message accelerator, business context streaming,
etc. without using more sophisticated (and expensive business server) for small
and/or middle size solutions.
The following picture shows an example from my previous article
Using Azure Lease Blob, where the business composition is done using an
EventStream
based on the Lease Blob.
As you can see, the above picture shows a composition of the Business Model
based on the EventStream
, where each WABS Bridge exclusively published an
Event
to the EventStream
, therefore each
Publisher can see a business context state for message acceleration, triggering
events, etc.
In the above introduction section I mentioned few my previous articles. Allow me to add one more article (WF4 Custom activities for message mediation) that will help you with the message mediation in the Workflow. This article will help you to build custom message mediators, invoker based on the Contract-First, Splitter, Analyzers, etc. in the Sequence and/or Flowchart fashion.
OK, let's continue with the concept and design. I am assuming you have a working knowledge of the Windows Azure Platform (BizTalk Services, Azure Blob, Service Bus, etc.)
Concept and Design
The concept of adding a Workflow into the WABS Bridge is based on the outbound WCF Custom Channel used in the message itinerary for generic destination WCF External Service Endpoint. Plug-in the WCF Custom Workflow Channel to the destination proxy enables the message exchange pattern - MEP (one-way or two-way) between the VETER and Workflow Pattern in sync or async manner.
The following picture shows an example of the simple one-way message pattern:
The outbound message from the VETER is forwarded to the One-Way
External Service Endpoint
configured for custom WorkflowChannel, see the
following configuration:
The message flows into the Workflow orchestration. As you can see in the above message itinerary, the message in the Workflow can be processed in the custom "second" VETER pattern orchestrated by workflow activities in the Sequence and/or Flowchart skeleton.
The WorkflowChannel has a responsibility to load and invoke a xaml workflow definition based on the query string located in the address Uri or message header. The source repository can be configured for default place such as a container in the Azure Blob (this article support only this feature) or folder in the project, for instance: Artifacts.
The query string contains at least one name-value pair such as the name of the xaml workflow definition. In addition, this is the place to configure loading of the xaml workflow definition, specifically by the request. The following are few examples of the query string:
xaml=LoopBack xaml=LoopBack&repository=bridges xaml=LoopBack&repository=bridges&settings=bridges/Settings/MySettings xaml=LoopBack&repository=bridges&ns=mynamespace& st=2014-01-18T20%3A00%3A51Z&se=2014-01-18T21%3A00%3A51Z&sr=c&sp=r&sig=yFriLn2xiJ...
Note, that the last example of the query string shows an example of the blob addressing with a SAS arguments.
Storing a xaml workflow definition in the Azure Blob is the default and the preferred way. The major reason for that is the blob metadata, where the workflow configuration can be stored. The following picture shows a data integration between the VETER and Workflow:
As you can see, the outbound VETER message is flowing to the Workflow via an
_inMessage
and returned back via an _outMessage
arguments. There
is a third mandatory workflow argument such as _metadata
. The
WorkflowChannel populated the input arguments such as _inMessage
and _metadata
arguments and based on the MEP fashion, it is
using an _outMessage
argument for Reply message. Also, if the xaml workflow resource is an Azure Blob
resource, then all blob's metadata are copied to the _metadata
argument.
Using the blob metadata for workflow configuration is a great feature, it
enables us to decompose the workflow variables in the business model and share them across
the
multiple workflows knows as Enterprise Variables. This is an advanced feature of
the custom WorkflowChannel, see more details about this scenario in my article
Azure
Virtual Bridge.
This article includes a full source code of the WCF Custom WorkflowChannel and it is very easy to modify loading of the xaml resource based on the requirements, for instance, a xaml definition from specific folder/database/blob/etc. and its metadata from other sharable (centralized) database/blob/folder/etc.
Custom message header - query
The custom message header query
is a string type qualified header for declaring a xaml workflow
definition such as name, location, etc. Every outbound message forwarded to the
WorkflowChannel Endpoint must have this header in the message.
The following snippet shows an example of the qualified header query
:
<query xmlns="urn:rkiss/2014/wf">xaml=Weather</query>
The header can also be created declaratively during a runtime in the VETER
Pattern or Route Action
.
The following screen snippet shows an example of the routing a message to the Weather workflow definition in the case of zip > 90000:
Implementation
The WCF custom channel
for Workflow has a very simple model supporting only IOutputChannel
and IRequestChannel
interfaces:
All custom logic related to the workflow target is encapsulated into the
custom WfHelper
static class, see the following code snippet:
public IAsyncResult BeginRequest(Message message, TimeSpan timeout, AsyncCallback callback, object state) { base.ThrowIfDisposedOrNotOpen(); base.RemoteAddress.ApplyTo(message); // load and create instance of workflow WorkflowApplication wfa = WfHelper.CreateWorkflowApplication(message, repositorySettingName, trackingParticipent); // run workflow asynchronously return wfa.BeginRun(timeout, callback, state); } public Message EndRequest(IAsyncResult result) { return WfHelper.EndRun(result); }
At this point, I would like to refer you to my previous articles where detail
description of the design and implementation of the
WorkflowChannel is described. There are few changes of the original WorkflowChannel made for WABS Bridge
are couple, such as
signing an assembly and adding a query header logic in the GetQuery
Method:
// query from header if (string.IsNullOrEmpty(query)) { int index = message.Headers.FindHeader("query", "urn:rkiss/2014/wf"); if (index >= 0) query = message.Headers.GetHeader<string>(index); }
Let's spend more time with Usage and Test of the
WorfklowChannel
in the WABS Bridge.
Usage and Test
First of all, the following are prerequisites (based on my development environment):
- Visual Studio 2012 Update 4
- Install Windows Azure SDK for .NET 2.2 (Visual Studio 2012)
- Windows Azure Platform account
- WABS namespace
- Http/s connectivity to the Internet
- Downloading package for this article
- Azure Storage Explorer
- Install Windows Azure BizTalk Service Explorer
- Knowledge and experience with Windows Azure Platform (Blobs, WABS, Service Bus, etc.)
In this section, I am going to show you step by step usage of the WorkflowChannel in the WABS Bridge.
Step 1. Download a package from this article
The following
screen snippet shows BizTalkServiceVirtualBridge
solution package included
in this article:
As you can see, the above solution includes 2 projects such as a WABS project -
BizTalkServiceVirtualBridge
and WokflowChannel
library
project. Basically for usage of the WorkflowChannel in your next WABS solution
only the WorkflowChannel.dll
assembly is required. Therefore,
the above WABS package is only for our test purposes of the WCF Custom
WorkflowChannel. As you can see, there is a folder Tools
with some assemblies used
for our testing. One more thing, the Artifacts folder has two xamlx resources
for our testing such as LoopBack
and Weather
. These
workflows are for testing purposes only.
Do not compile this solution before the step 2, You have to update the solution with your accounts.
Step 2. Modify solution for your accounts.
The solution must be modified for your accounts such as BizTalk Service
URL
property and configuration file for all following destination
endpoints:
As I mentioned earlier, the WorkflowChannel is using an Azure Blob Storage
for repository of the xaml workflow definitions. For test purposes, this
solution uses (like it is shown in the above config) two containers, one is for
assemblies
and the other one is bridges
for xaml
resources. Please, create those private containers in your blob storage.
Once these changes are made, the solution is ready for deployment to your WABS endpoint.
One more help, if you will have an issue with building this solution, please close the Visual Studio and reopen it again, but use only the Build feature. I believe there is a bug in the Visual Studio 2012 when holding assembly reference. That's my experience and I hope that upcoming WABS for Visual Studio 2013 will resolve it.
Step 3. Deployment
To deploy WABS solution from the VS2012, the following dialog prompt will be show up on your screen:
The deployment Endpoint will show your WABS namespace and you must fill in
the AcsNamespace
and SharedSecret
properties. After that, you can watch the
deployment process in the Ouput
panel and I hope without any errors.
If the deployment succeeded, you should see the following in the Server Explorer this solution in your WABS namespace:
The
Windows Azure BizTalk Service Explorer is a great tool. It allows to refresh, restart, update, etc. your WABS.
Also, you can see a list of the uploaded assemblies. The above picture shows those
assemblies which we need to have for our next testing. Note, that the missing
assemblies can be manually uploaded or added in the solution reference with the
flag True for Copy Local
property. By the way, this flag is useful for
controlling an assembly upload, it will speed up the re-deployment process
when the bridge itinerary has changed, for instance.
Now we can focus on Workflow part.
Before we proceed, let's pre-test for our deployed WABS which will indicate if we are on the right path to the Workflow.
Let's invoke some Workflow, at this moment we can invoke any kind of name because our blob storage is empty.
Step 4. Invoking a non-existing Workflow
For easy way to send a message to the WABS, we can use my "dirty tool" included in the solution folder Tool.
Please, launch the MessageSender
program and populate your account values for
your WABS namespace. Setup a query string value for xaml=ABCD
. After that,
press the button Send. You should get a response message from your WABS
indicated the following:
That's a good sign, that your WABS should work in the following steps.
Step 5. Invoking an existing Workflow
In this step we are going to upload blobs to your Azure Storage. Note, you will need to use your own credit for WABS and Blob Storage.
Any 3rd party Azure Storage Explorer, but in this step let's used very popular and free Azure Storage Explorer from my friend David Pallmann.
Please, upload the following xaml resources to your Blob Storage container
bridges
:
- Artifacts/LoopBack.xamlx
- Artifacts/Weather.xamlx
make the following changes:
- delete an extension
.xamlx
in the name of the blob (this is an option, but in this case we have to use a full name of the xaml resource in the query string) - use content type
application/xaml+xm
(this is an option in the opposite typetext/plain
, where we can redirect a resource to another blob) - add the following metadata for Weather blob:
Please, upload the following assemblies to your Blob Storage container
assemblies
:
- Tools/MessageMediationActivityLibrary.dll
make the following changes:
- delete an extension
.dll
in the name of the blob
In the case of using a custom activities in your Workflow, you have to upload its assembly into this container and restarted a WABS be sure that this assembly is going to load into the WABS Bridge process (web role).
That's all. Now it is the time to do some testing. Let's start with something very simple and using only a Microsoft Workflow Activities from the .Net 4.5, some simple workflow such as a LoopBack.
Step 6. Test LoopBack Workflow (Router)
This test is using a Router WABS Bridge. Its address is
https://your_namespace.biztalk.windows.net/default/Router
Any query string will be forwarded into the WorkflowInvoker
destination with
a WCF custom WorkflowChannel.
Use the MessageSender program to send a request message with a query header:
<query xmlns="urn:rkiss/2014/wf">xaml=LoopBack</query>
The following screen snippet shows a response from the LoopBack WABS Bridge:
You can work with different zip code
in the request message payload to see the changes in the response message. The
response panel shows a TrackingID
. This id
can be used
for tracking a message through the Router Bridge using the WABS Explorer or
Azure Portal.
Note, that this Router Bridge can be used for invoking any xaml workflow
definition stored in the container bridges
. I am sure, you will use
it to invoke your workflows as well.
Step 6. Test Weather Workflow (WeatherBridge)
This test will demonstrate the capability of the VETER bridge to select two
different xaml workflow definitions from the Azure Blob Storage. The Weather
Workflow is a very straightforward workflow using a custom activity
CreateMessage
(from my recommended article
WF4 Custom activities for message mediation) to create a request and response
message from/to call to the Weather public service. In this test we are testing
the capability of the loading a custom assembly from the container assemblies
.
The following picture shows a WeatherBridge
with two destinations configured
for WorkflowChannel:
Note, that the inbound message doesn't need to have a query header, because this header is added in the outbound message based on the filter.
As you can see, based on the payload value ZIP, the message is routing to the properly destination. There are 3 groups of the filters. The third one is when ZIP = 90000. In this case, the caller will receive an error message.
The following screen snippets show these cases:
LoopBack Workflow:
Weather Workflow:
As you can see, there is a custom header in the last response message configured in the weather's metadata. You can use this configuration during your testing without re-deploying the bridge or updating blob content.
That's all for testing. I hope both tests succeeded on your WABS deployment. In addition, the demonstration showed you the capability of the VETEW Pattern using the WorkflowChannel.
Advanced Tooling
My article RoutingService on Azure is described a small modification of the Azure Storage Explorer which will associate a program with blob content. The following picture shows this user interface extension:
Clicking on the Edit
link, the re-hosted WorkflowDesignerTester
will launch for editing
of
the xaml definition. The changes in the xaml can be saved in the blob. Note,
that the fresh version of the WorkflowDesignerTester
program can be found in the
solution Tools folder.
Conclusion
This article described how we easily we can extend the pre-built VETER Pattern in the Windows Azure BizTalk Service (WABS) by Workflow (WF) capability. Using a WCF Custom WorkflowChannel in the Destination Endpoint, the outbound VETER message pipeline continues in the Sequence and/or Flowchart skeleton of the Workflow. This article described a new concept pattern such as VETER + WORKFLOW = VETEW. I hope you enjoyed it.
References and useful links:
[1] Windows Azure
[2] Windows Azure SDK 2.2 for .Net
[3] Windows Azure BizTalk Service Explorer
[5] Windows Azure: General Availability Release of BizTalk Services, ...
[7] Walkthrough of New Windows Azure BizTalk Services
[10] WorkflowChannel for Routing Service.
[11] Windows Azure BizTalk Services: EDI Capabilities
[12] Developing custom bridge components for Windows Azure BizTalk Bridges
[13] BizTalk360 Blog
[14] Extending Windows Azure BizTalk Services
[15] Securing Azure BizTalk Services using ADFS
[16] Extending Windows Azure BizTalk Services Using Message Inspectors