Click here to Skip to main content
14,930,087 members
Articles / General Programming / Architecture
Posted 5 Jun 2017


8 bookmarked

A Framework Implementation using Pipe and Filter Architecture

Rate me:
Please Sign up or sign in to vote.
4.27/5 (9 votes)
6 Jun 2017CPOL5 min read
Basic concept of pipe and filter; what is pipe and filter architectural style; when you need to use it; framework of the pipe and filter pattern; a tiny AI tool implementation using it

Pipes & Filters Architectural Pattern

Readers of the Topic - Developers, Architects

Coverage Topics

  • Basic Concept of Pipe and Filter
  • Pipe and Filter Architectural Style
  • Basic Pipe & Filter Example
  • When Do You Need to Use It
  • Pipe and Filter Framework
  • A Tiny AI Tool Implementation

Let's Drill Down the Basic Concept

Image 1

A mineral water manufacturing factory have a water source. But it contains virus, bacteria, colors, salt, sand, stone, and grass. So, they need a water treatment to clean the water.

Image 2

Solution: They can add few filters to clean up the dirt water through the pipe-lines. Therefore, they will be able to reserve the mineral water to the water tank.

Image 3


Image 4

A filter has an input pipe and an output pipe.

Pipes and Filters

A filter is connected by pipelines and the output of one filter is the input to the next filter.

Image 5

Pipes and Filters Architectural Style

It has two subsystems and that is pipe and filter.

  • Filter: Filter processes the data.
  • Pipe: Pipe or pipeline is a connector between two filters.

When Do You Need to Use It

If you have continuous data flow which needs a chain of processes.

Implementing an AI Tool using Pipe and Filter Architectural Pattern

The Problem

An organization needs to verify the education history of their employees. Say, I'm the employee of that organization and I have finished my bachelors and master's degree in computer science. The organization will consider my highest category of the degree. I mean, I have 2 degrees, bachelors and masters. The masters is the highest degree compared to bachelors. Now they will verify the master's degree. In this case, bachelor degree will be ignored.

Therefore, they need a tool which will find out the highest achieved degrees. It has two parts:

  • Classify the degree Category
  • Pick up the highest achieved degree only.

Classify the Degree Category

Image 6

If employees have degrees such as Bachelors of Science and Masters of Science, then:

  • "Bachelors of Science" is in category of bachelor, and
  • "Masters of Science" is in category of master

Process Data to Filter out Highest Achieved Degree

  • Find out the possible category from the candidates degrees
  • Filter out the highest category and rank from that degree

Basic Concept of a Tiny AI Tool

Initial Data and Data Source

Data source - Dictionary

Taking 2 data source dictionaries are as follows:

  1. Secondary words dictionary: The word itself does not have any meaning but it will help to make a sentence. So, I'm guessing, 1 points for these words. The table is given below.
  2. Primary key words dictionary: This keyword will help to determine the category of the degree. I mean- if we know that the key word is bachelor or master or diploma, then it will make sense that we can determine the degree. So, I'm guessing 5 points for those words.

Image 7

Class Diagram of Core Interface of Pipe and Filter

Image 8

Implementation of Core Interface

The interfaces uses generic type. So, define your own data input models as well as sink model. We need to use it into the filter classes.

Image 9

Filters Implementation

Candidate-filter class will inherit the FilterBase class and we have to implement only single method which is called the process. Now we can implement as many filters as we need.

Image 10

Data Passing to the Channel of the Framework

  • Initialize the following objects, such as, input data model, SingleTemplateChannel and MainChannelStaters classes.
  • Next, initialize ModelForSetupFilters model class and inject the required filters into it, then send it to the singleChannel. SingleChannel will register all of the filters from ModelForSetupFilters.
  • Setup the mainChannelsStartup object which is given below: channelstaters.SetupChanelList(singleChannel, educationDegreeList);
  • Finally, call the starter and get the final results from resultList objects. IList resultList = Channelstaters.ChannelCaller();

Image 11

The Parallel Process of the Channels

We can include as many as filters and channels that we need. Each of the inputs will pass through the Channel and each of the channels will hold a list of filters. Say, if we have five education histories - that means, we need 5 channels and it will run simultaneously for parallel processes.

Image 12

Example to Find out Highest Achieved Degree

Input - Output

Input: taking a bad input: Master of Bachelor Science bla bla

Expected category: Bachelor

Data Processing Sequences from the Filters

DegreeMatchFilter: Finding the keywords

A bad input: Master of Bachelor Science bla bla :(

Clean-Keyword: Master Bachelor Science


From the Keyword dictionary, the point for each words:

Image 13

Step 1: Matching Text from the Data-Source

Result depends on the keywords: Master; Bachelor; Science

[0]: "Master of Science"

[1]: "Master's Degree"

[2]: "Master of Business Administration"

[3]: "Master"

[4]: "Master of Arts"

[5]: "Master's"

[6]: "Bachelor of Science"

[7]: "Bachelor of Arts"

[8]: "Bachelor's Degree"

[9]: "Bachelor"

[10]: "Bachelor's"

[11]: "Bachelor of Business Administration"

[12]: "Bachelor's Degree (±16 years)"

[13]: "Bachelor's of Science"

[14]: "Bachelor of Science in Nursing"

[15]: "Bachelor of Science"

[16]: "Bachelors of Science"

[17]: "Master of Science"

[18]: "Associate of Science"

[19]: "Associate of Applied Science"

[20]: "Masters of Science"

[21]: "Bachelor's of Science"

[22]: "Bachelor of Science in Nursing"

[23]: "associates of science"

Step 2: Count the Points for Each Sentence

Image 14

Step 3: Match and Remove the Sentences Comparing With Max Point

According to the table of step 3, the maximum point is 6. So, remove all of the sentences which are less than 6.

Image 15

Removing the duplicate sentences and the output from these steps are:

Image 16

DegreeCategoryFilter : Find out the category

Image 17

DegreeRankFilter: Find Rank Point

Image 18

DegreeCategoryProbabilityPartAFilter: Applying Probability


Image 19

Image 20

DecisionMakingInputFilter: Remove Category and Rank which is Lower

Find out the category and pank points. According to the above table, the maximum point is 13.5. So, removing all of the sentences which are less than 13.5.

FinalDecisionMakingOutputFilter: Final Output

The max point is 13.5 and the category is bachelor.

Therefore, the output: Bachelor of Science :==> Points [13.5]; Bachelors :=> Rank 4

So, bachelor is the highest achieved degree that needs to verify.

Category: Bachelor; Rank: 4

Demo Time!!

Image 21

We never know where time is taking us. Don't just keep your eyes on the code only. Look back at your design. It shouldn’t be the victim of the worst nightmare.

Find the demo and full source code of the project (attached).


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Habibur Rony
United States United States
Lazy software engineer, don’t believe in hard work.

Comments and Discussions

-- There are no messages in this forum --