RESTGrid

• Source Code on GitHub - http://github.com/WorkMaze/RESTGrid
• Orchestration Engine docker container for MySql provider http://hub.docker.com/r/workmaze/restgrid.mysqlengine.workflow/
• REST API docker container for MySql provider http://hub.docker.com/r/workmaze/restgrid.mysqlengine.api/
• Orchestration Engine docker container for AWS DynamoDB (NoSQL service by AWS) https://hub.docker.com/r/workmaze/restgrid.dynamodbengine.workflow/
• REST API docker container for AWS DynamoDB provider http://hub.docker.com/r/workmaze/restgrid.dynamodbengine.api/

Introduction

RESTGrid is a simple Workflow\ETL system in .NET Core which uses REST services to interact with the outside world. It has the following features:

• The system has the ability to define workflows using a JSON format.
• The workflows can be synchronous or long running & asynchronous.
• Each step in the workflow is a call to a REST service.
• Steps can be synchronous wherein the next step is called after the previous step is executed.
• Steps can also be asynchronous wherein after the execution of a step, the workflow goes into a waiting stage and waits for an external call.
• A workflow can be started, re-started using a REST API.
• Administration using a REST API.
• Ability to define JSON transformations using JUST - http://www.codeproject.com/Articles/1187172/JUST-JSON-Under-Simple-Transformation.
• Interface based system and hence can be hooked to different data sources.
• Docker containers available for MySql provider.
• Docker containers available for AWS DynamoDB (NoSQL service by AWS) provider.

Background

One of my previous articles (https://www.codeproject.com/Articles/1202556/FloatingBridge) focuses upon a Messaging\ETL\Workflow system based on .NET Framework & MySql.

My intention with this software was to provide a simpler alternative to the above solution. This does not include the rich windows UI features. However, there are some additional features:

• The source code is in .NET Core & docker files included in the source.
• Docker containers for the MySql provider are available at Docker Hub.
• Although the rich windows UI features are missing, a REST API for administration is included.
• At the moment, only the MySql provider is included. However, one can easily develop a new provider using the interfaces provided.
• The provider for the NoSQL service by AWS (DynamoDB) has also been included in the solution now.

System Architecture

A brief description about the various components of the system. Below is a schematic diagram representing the system architecture:

Orchestration Engine

This is the core of the system which enqueues tasks, reads the business logic of the workflow and runs tasks.

REST API

This is used by external systems to start\re-start workflows as well as users to administer\set-up workflows.

Backend

This is the store where the configuration as well as historical data is stored. RESTGrid is the main project which defines the interfaces, objects and the orchestration logic for the system.

Since this is an interface based project, we can hook up any backend as long as we implement the interfaces defined.

The core library of RESTGrid is available as a Nuget package: https://www.nuget.org/packages/RESTGrid/

Business Logic

The business logic of the system is defined using a standard JSON format. The format is very similar to FloatingBridge. However, there are some omissions to make it simpler.

The root object contains the following two properties:

• Start (the starting task for the workflow)
• Tasks (An array of all the subsequent tasks in the workflow)

Task

The Task JSON contains the following properties:

• Identifier - A unique identifier which identifies the task
• Type - The type of task
• Next - An array of task identifiers pointing to the tasks that could be run after this task
• TaskRetries - The number of times a task can be retried before it fails
• TaskProperties -The JSON object that contains information on how the task should be run (can be transformed using JUST)
• RunCondition - The condition which must be satisfied for the task to run

Run Condition

The run condition decides the condition which needs to be satisfied for a task to run. It has the following properties:

• Evaluator - An expression that needs to be evaluated (can be transformed using JUST).
• Evaluated - An expression to be evaluated against (can be transformed using JUST).
• Operator - The standard operators:
  1. stringequals
  2. stringcontains
  3. mathequals
  4. mathgreaterthan
  5. mathlessthan
  6. mathgreaterthanorequalto
  7. mathlessthanorequalto

Task Types

A task can be as one of the following four types:

• Sync - Synchronous REST calls (next step is executed after a synchronous call).
• Async - Asynchronous REST calls (workflow waits for an external input after executing an asynchronous call).
• Transformer -Transforms the message body (a JUST transformation)
• Splitter - Splits the message body based on an array inside the JSON (a JUST split)

Task Properties for REST tasks (Sync & Async)

The JSON for task properties of REST tasks contains the following properties:

• Url - Url of the REST Service
• Method - GET, POST, PUT or DELETE
• Headers - Key-Value pair JSON (optional)
• Body
• QueryString

Task Properties for Transformer

• TransformerID - The integer ID identifying the transformer JSON (In case of MySql, this is the primary key of the transformer table)

Task Properties for Splitter

• ArrayPath - The JSONPath pointing to the array.

Example of a Business Logic JSON

"Start": {
    "Identifier": "CreateUser",
    "Next": [
     "CreateRole"
    ],
    "RunCondition": null,
    "Type": "Transformer",
    "TaskProperties": {
     "TransformerID": "3"
    }
 },
 "Tasks": [
    {
     "Identifier": "CreateRole",
     "Next": [
       "AddApplication",
       "Notify"
     ],
     "Type": "Splitter",
     "TaskProperties": {
       "ArrayPath": "$.Organization.Employee"
     }
    },
    {
     "Identifier": "AddApplication",
     "Type": "Sync",
     "Next": [
       "Approve"
     ],
     "RunCondition": {
       "Evaluated": "CreditCard",       
                    "Evaluator": "#valueof($.MessageBodyJson.Organization.Employee.PaymentMode)",
       "Operator": "stringequals"
     },
     "TaskProperties": {
       "Url": "http://localhost:5001/",
       "Method": "POST",
       "Headers": null,
       "Body": "#valueof($.MessageBodyJson.Organization.Employee.Details)",
       "QueryString": "api/table/user"
     },
     "TaskRetries": 0
    },
    {
     "Identifier": "Notify",
     "Next": [
       "Approve"
     ],
     "Type": "Async",
     "RunCondition": {
       "Evaluated": "Cash",
       "Evaluator": "#valueof($.MessageBodyJson.Organization.Employee.PaymentMode)",
       "Operator": "stringequals"
     },
     "TaskProperties": {
       "Url": "http://localhost:5001/",
       "Method": "POST",
       "Headers": null,
       "Body": "#valueof($.MessageBodyJson.Organization.Employee.Details)",
       "QueryString": "api/table/user"
     },
     "TaskRetries": 0
    },
    {
     "Identifier": "Approve",
     "Type": "Sync",
     "TaskProperties": {
       "Url": "http://localhost:5001/",
       "Method": "POST",
       "Headers": null,
       "Body": {
         "Message": "Your payment has been approved"
       },
       "QueryString": "api/table/user"
     },
     "TaskRetries": 0
    }
 ]
}

The above business logic JSON represents a workflow which would look like the schematic diagram below:

Using the Code

RESTGrid is an interface based system. The core library provides interfaces which can be inherited to implement your own backend providers.

The following interfaces are provided:

IAdministration

namespace RESTGrid.Interfaces
{
 public interface IAdministration
    {
        void CreateWorkflowType(string workflowTypeName, JObject businessLogicJson);

        void CreateTransformer(JObject transformerJson);

        WorkflowHistory GetHistory(string workflowID);
    }
}

IOrchestration

namespace RESTGrid.Interfaces
{
    public interface IOrchestration
    {
  
        void PublishWorkflowStep(string workflowTypeName, Guid workflowID, 
           JObject messageBodyJson, JObject customPropertiesJson, string stepIdentifier,
    bool stepSucceeded, bool workflowCompleted, int retries, bool active, 
           string runStepIdentifier, string splitID);

        void SetWorkflowActive(JObject messageBodyJson, string customPropertyName, 
             string customPropertyValue);

        List<RESTGrid.Models.Queue> Enqueue();

        JObject GetTransformer(int transformerID);
    }
}

Once you implement the IOrchestration interface, you can easily develop your own Orchestration\Workflow engine. Here is how the MySql engine is implemented:

MySqlOrchestration orchestration = new MySqlOrchestration(connectionString);
OrchestrationEngine engine = new OrchestrationEngine(orchestration);
Console.WriteLine("Running orchestration engine...");
while (true)
{                   
	engine.Run();
}

The REST API for MySql provider has the following APIs:

Administration API

GET {url}/api/Administration/History{workflowID}
Returns 200 OK with a JSON containing the entire workflow history.

POST {url}/api/Administration/WorkflowType/{workflowTypeName}
Body containing the Business Logic JSON.
Returns 204 No Content.

POST {url}/api/Administration/Transformer
Returns 204 No Content.

Orchestration API

PUT {url}/api/Orchestration/Workflow/{customPropertyName}/{customPropertyValue}
Body (Optional) - <JSON containing the new input message>.
Returns 204 No Content.

POST {url}/api/Orchestration/Workflow/{workflowTypeName}
Body containing the JSON message in the following format:-
{
"MessageBody": <JSON containing the input message>,
"CustomProperties": <One-level JSON representing a key value pair>,
}
Returns 204 No Content

Points of Interest

• Since this project uses .NET Core & docker containers, it can be easily hosted on the cloud providers which have support for docker.
• The MySql database can either be a standalone system or can be hosted on the cloud providers which have support for it.
• The REST services which are called by the system can be hosted in the cloud making it possible to host the entire system on the cloud.

History

1. The first version of RESTGrid (MySql provider for RESTGrid)
2. Added the AWS DynamoDB (NoSQL service by AWS) provider