Click here to Skip to main content
Click here to Skip to main content

Implementing a batch system in BizTalk in order to keep two applications/systems in sync.

, 9 Apr 2013 CPOL
Rate this:
Please Sign up or sign in to vote.
This article talks about implementing a batch system in BizTalk in order to keep two applications/systems in sync.

Setting the Context

Batch applications are still critical in most organizations in large part because many common business processes are susceptible to batch processing. There are certain scenarios when one would need to design a batch system, in order to handle and/or control the way he data flows into a system.

Even with advances in batch program development, problems with batch performance are still very common. This is particularly painful during implementations. Operations team needs to ensure that the batch window is not breached, to guarantee their Service-Level Agreement (SLA) is met.

Batch processing is also used for efficient bulk database updates and automated transaction processing, as contrasted to interactive online transaction processing (OLTP) applications. This article attempts to explain the implementation of a non trivial batch system using BizTalk.

Phase - I - Building the Batch system

A batch generally consists of a batch table and a configuration table. The batch table would contain the primary key of the record and a status flag. The batch table holds the status of the record in the batch. The batch is considered complete when all records are either in SUCCESS or FAILED status. The following is the list of status values used in the batch.

Step 1: Creating the Batch table

The batch table consists of a unique key to identify a batch record and a related status with it. In this case we are trying to create a batch table for an Account CRM entity as an example. A tiny database with few tables are created on the CRM system. See the figure for a clear understanding of it.

Step 2: Creating the Configuration table

The Configuration table, in this case Batch Config consists of a key which holds the last batch run date/time. This would prevent the re-run of a batch record to avoid duplicates.

Step 3: Building the SQL Batch Job

The SQL batch job would fill in the batch table with the relevant records matching a given criteria. The sample script for the batch job is shown.

Phase - II - Configuring BizTalk to process the Batch

Step 1: BizTalk Receive location

The BizTalk receive location can be used to pull in the batch into BizTalk. The BizTalk receive location configuration is shown below.

Step 2: BizTalk Orchestration

Once a batch record is within BizTalk, an orchestration can be written to process the record and perform various operations on it. The Batch orchestration code is beyond the scope of this article.

Step 3: BizTalk Status Flow

The status flow lifecycle is shown in the figure. The initial status is LOADED and this could end with either a SUCCESS or FAILURE. Custom logic can be written within the Orchestration to move the status to DEFERRED.

Step 4: System Architecture

The overall system architecture is depicted in the diagram below. Note that for pushing data into CRM, you might want to use the CRM web service using the BizTalk WCF adapter.

Phase III - Further action items

As a developer and designer of BizTalk orchestrations, you must determine what additional actions are required to be taken by the orchestration. This could involve the following

  • Posting a message to a Web Service.
  • Sending an email or an Acknowledgement.
  • Updating a record in a table.

Note that a similar set of actions are required for the two-way sync process.

Further references on Batch processing

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Naveen Karamchetti
Architect AT&T Wi-Fi Services
United States United States
Naveen has done his Masters (M.S.) in Computer science, has started his career programming the mainframes and now has more than a decade of programming, development and design experience. Naveen has a sharp eye and keen observation skills. Naveen has worked for several companies and strived hard to build large scale business applications and bringing better solutions to the table.
Quite recently Naveen has built a fairly complex integration platform for a large bank. His hobbies include training, mentoring and research. Naveen spends his free time visiting National Parks nationwide.
 
Naveen has developed the BizTalk Control Center (BCC)
http://biztalkcontrolcenter.codeplex.com

Comments and Discussions

 
-- There are no messages in this forum --
| Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.141223.1 | Last Updated 9 Apr 2013
Article Copyright 2013 by Naveen Karamchetti
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid