CMS/TSO PipeLines for Windows - Part I Syntax

DownLoad PC-Pipes Application library

Introduction

CMS/TSO PipeLines is a product that runs on IBM mainframes. On a z/VM system it runs under CMS. On a z/OS system it runs under TSO. As the name suggests data is passed through a series of builtin or user defined stages. Along the way the data can be modified or deleted from the pipe. You may be saying to yourself there is nothing new in that. What is different is that PipeLines supports concurrent parallel pipes and data can be routed into and out of a pipe.

System Requirements

While the PipeLib class library can be accessed from a .Net program, I think you will find that running it under PowerShell more useful. I do all of my testing through PowerShell and any pipelines I supply will most likely be PowerShell scripts. Therefor as minimum system requirements I recommend the following.
PowerShell V1.0
.Net Framework V2

Syntax

To begin I what to describe the syntax diagrams that are used in IBM's manuals and in this product. The base of a syntax diagram is the command line. The elements of a command line are:

>>—————><	A command line begins with >> and ends with ><.
—————>	This indicates the syntax is continued below.
>—————	This indicates that this is a continuation from above.

An option that appears directly on the command is required. It MUST be supplied.

>>——Command——required option———————————><

If items are stacked below a required option, this indicates that one of the items MUST be selected.

>>——Command——┬option1───┬———————><
             ├option2───┤
             └option3───┘

If an item appears below the command line, then it is an optional and need not be supplied.

>>——Command——┬—————————─┬———————><
             ├option1───┤
             └option2───┘

If an item appears above the command line, then it is the default if no option is supplied.

             ┌default—──┐
>>——Command——┼─────————─┼———————><
             ├option1───┤
             └option2───┘

An arch above an option indicates it can be entered multiple times or that multiple options of a group may be selected. Below indicates that you can enter option1 and/or option2 and the default is assumed if neither is supplied. If a delimiter is needed between options it is specified after the < on the arch.

            ┌─<────────┐
            │┌default—┐│
>>——Command—┴┼────————┼┴——————><
             ├option1─┤
             └option2─┘

A syntax segment or group is used when a portion of the syntax is repeated or recursive. It can also be used to simplify the main line of the command and break it into seperate command lines.

>>——Command—┤segment├———><

Group segment
├───segment options────┤

An option that appears entirely in lowercase indicates a user supplied variable. If the option is entirely or starts with uppercase then it is a keyword. The uppercased characters indicate the minimum abbreviation of a keyword.

>>——Command——KEYword——variable——————><

In the above, the keyword could be entered as just "KEY" and "variable" is some user supplied data.

Lets consider a real world case. Below is the syntax diagram of the SQL builtin stage.

       ┌─<───────────────┐
       │┌NOPAD COLSEP ~─┐│
>>─Sql─┴┼───────────────┼┴┬─────────┬─><
        ├┬PAD───┬───────┤ └┤connect├┘
        │└NOPAD─┘       │
        └COLSEP xorc────┘

Group Connect

         (1)        ┌─<─;──────┐
├─CONNect──┬SQL────┬┴key=value─┴─┤
           ├ODBC───┤
           ├OLE────┤
           └ORACLE─┘

Group Request

├─┬────────────────────┬┬┬─────────┬SELECT─select statement─┬─┤
  │  ┌─<────────────┐  ││└DESCribe─┘                        │
  └(─┴┬────────────┬┴)─┘└DESCribe─tablename─────────────────┘
      ├┬PAD───────┬┤
      │└NOPAD─────┘│
      └COLSEP xorc─┘

Notes
1. When a Connect statement is specified on a Sql stage specification, 
   omit the "Connect" keyword.

Points of interest:

1. The rules for appearing on or below a command line also apply to option lines. Note the "PAD", "NOPAD" options, one or the other may be entered but not both. if you omit the options then the default of "NOPAD COLSEP ~" is assumed
2. You can optionally supply a "Connect" statement on the stage specification. If you do the according to the note you omit the "CONNECT" keyword.
3. On a "Connect" statement you select the connection type and specify the "key=value" pairs that define a connection delimited by a semicolon.
4. The "Group Request" is not a part of the syntax of the SQL stage. It describes an input record read by the stage. On a Request you can override the options specified on the stage specification and the options must be enclosed in parentheses. You can specify a SQL select statement, optionally preceded by the "DESCRIBE" keyword. Or you can specify the "DESCRIBE" keyword followed by a table name.

PipeLine Definition Syntax

Below is the syntax diagram for a pipeline definition. The syntax is further explained thru examples that follow.

                                          ┌─<─endchar┐
>>──┬─────────────────────────┬┬─────────┬┴┤PipeLine├┴─><
    │  ┌─<─────────────────┐  │├stagesep─┤
    └(─┴┬┬STAGESEP──┬xorc─┬┴)─┘└endchar──┘
        │└SEPerator─┘     │
        ├ENDchar xorc─────┤
        ├ESCape xorc──────┤
        ├NAME─pipename────┤
        └TRACE────────────┘

Group PipeLine

  ┌─<─stagesep───────────────────────────────────────────────┐
├─┴┬────────────────────────────────────────────────────────┬┴─┤
   ├┬────────────┬┬───────────────┬stagename─┬─────────────┬┤
   │└┤LabelGroup├┘└(─┬───┬TRACE─)─┘          └stageoptions─┘│
   │                 └NO─┘                                  │
   └┤LabelGroup├────────────────────────────────────────────┘

Group LabelGroup

├─┬────────────────┬─┤
  ├label:──────────┤
  └label.streamid:─┘

• Options

STAGESEP xorc
SEPerator

Specify the character to be used to seperate the stages of a pipe. It can be specified as a single character or as the hex digits of a character. The default seperator is the vertical bar, "|".

ENDchar xorc

Specify the character that separates the pipes of a pipeline. It can be specified as a single character or as the hex digits of a character. There is no default.

ESCape xorc

Define an escape character that can be used to override the processing of a character that has special meaning to the PipeLine parser. Any character that folows the escape character is treated as an ordinary character. There is no default.

NAME pipename

Specify a name to be associated with this pipeline.

TRACE

Turn on tracing for the pipeline.

• Pipeline

stagesep

The stages of a pipe are separated by the STAGESEP character. The default is "|".

endchar

If there are multiply pipes in the pipeline, seperate them by the ENDCHAR.

NO

If tracing is turned on for the pipeline then "NO TRACE" will turn it off for this stage of the pipeline.

TRACE

Turn tracing on for this stage of a pipeline. If prefiexed by the "NO" keyword then tracing is turned off for this stage.

stagename

The name of the stage to be executed.

stageoptions

Supply any options needed by the specified stagename.

label:
label.streamid:

A label that identifies additional input and/or output data streams for the stage it is specified on. The first occurrence of a label is called a label definition. It establishes a connection point where other stages, in other pipes, receive data from or send data to. Each subsequent use of the same label is called a label reference. Use label references to define additional input and output streams for the stage. To use a label reference, specify a stage containing only a label with no stage name.

A streamid assigns a symbolic name to a stream. Name a stream by adding an identifier to the label. Write the label immediately followed by a period (.) and up to 4 alphabetic characters or a combination of alphabetic characters and digits that includes at least one alphabetic character. A stream identifier must be immediately followed by a colon with no intervening blanks.

Sample PipeLines

In its simplest form a pipeline consists of a series of stage definitions delimited by the STAGESEP character.

 stage1
|stage2
|...
|stagen

The next step up in complexity would be defining multiple pipes in one pipeline.

(end ?) stage1-p1
   |...
   |stagen-p1
 ?  stage1-p2
   |stage2-p2
   |...
   |stagen-p2

Here we specify a "ENDCHAR" that seperates the pipes in a pipeline definition.
But the full power of pipelines is only realized when you use stream labels.

A word about data streams. Every stage in a pipe has two streams by default, the primary input and output streams. The primary input reads data from the previous stage in the pipe and the primary output writes data to the next stage in the pipe. Just because a stream is defined does not mean it is connected to anything. For example, the first stage in a pipe has a primary input but it is not connected because there is no previous stage to connect to. Simiraly the last stage of a pipe has a primary output but it is not connected.

A stage that defines a stream label defines a connection point for additional input or output streams. A stream label can be referenced in one of three positions in a pipe.

1. The first stage if a pipe.
   A stream label referenced by the first stage of pipe means that an output stream is created in the defining stage. The stage that defined the label will write data to the stage following the referencing label.
2. In the middle of a pipe.
   A stream label referenced in the middle of a pipe creates input and output streams in the defining stage. Data from the stage previous to the refernecing label is written to the stage that defined the stream label. The defining stage in turn will write data to the stage that follows the referencing label.
3. The last stage of a pipe.
   A stream label referenced by the last stage of pipe means that an input data stream is created in the defining stage. Data from the stage previous to the referencing label is written to the defining stage.

Also note that a stream label can be referenced more than once. Each reference creates a new input or output data stream. The data streams are named and numbered as follows:

Name	Number
Primary	0
Secondary	1
Tertiary	2
Quarternary	3
Quinary	4
Senary	5

Consider the follow pipeline segment

1. '(end ?) < file.txt'
2. '|l:find abc'
3. '|...'
4. '|f:faninany'
5. '|console'
6. '? l:'
7. '|...'
8. '|f:'

What this pipeline does is:

1. Define the ENDCHAR and read the contents of the file "file.txt"
2. Define the stream label l: and execute the find stage.
   This stage looks for the string abc starting in column 1 of a record. If the string is found it passes the record to the next stage in the pipe. If the string is not found then the record is wriiten to the data stream defined by l:, if it is connected.
3. Do something with the records that contain abc starting in column 1.
4. Define the stream label f: and execute the faninany stage. This stage read records from the previous stage in the pipe and any connected data streams.
5. Execute the console stage. As the name implies it writes data to the terminal screen.
6. We start a new pipe and reference the stream label l:. Since the stream label comes at the start of a pipe definition it creates the Secondary output stream for the find stage on line 2. The records not selected by the find stage come here.
   If later on in the pipeline we referenced the l: label again, then we would create the Tertiary output stream for the find stage. Or, depending on where the reference occured, we could create a Secondary input data stream for the find stage. However the find stage only supports a Secondary output stream.
7. Do something with the records.
8. We reference the stream label f:. Since it is a the end of a pipe definition it defines an Secondary input data stream on the defining stage. It sends data from the previous stage to the stage that defined the stream label. In this case the faninany stage on line 4. Unlike the find stage the faninany stage supports multiple input streams. If later on in the pipe we reference the f: label again we would create the Tertiary input stream.

A Real World Example

This stage creates a HTML table of the builtin stages and if there is documentation for the stage it creates a link to it. Note the definition below appears as it would in a PowerShell script.

1. '(end ?) shell ls ..\..\runtime\* -include *.cs -name '+
2. '-exclude util.cs,stageprocessor.cs,dispatcher.cs,addcallstream.cs,spec*,codepages.cs,'+
3. 'old*,parsercntrl.cs,structCntrl.cs'+
4. '|nlocate /File.cs/'+
5. '|chop .'+
6. '|m:faninany'+
7. '|sort'+
8. '|l:lookup detail'+
9. '|spec $<a href="Doc/$ 1 1-* n $.html">$ n 1-* n $</a>$ n'+
10. '|f:faninany'+
11. '|literal <a href="Doc/AppendFile.html">>></a>'+
12. '|literal <a href="Doc/CreateFile.html">></a>'+
13. '|literal <a href="Doc/ReadFile.html"><</a>'+
14. '|spec $<td>$ 1 1-* n $</td>$ n'+
15. '|join 7'+
16. '|spec $<tr>$ 1 1-* n $</tr>$ n'+
17. '|literal <table cellspacing="5" cellpadding="5">'+
18. '|literal append +</table>+'+
19. '|> StageTable.html'+
20. '? literal Hostbyaddr Hostbyname NFind StrFind StrFrLabel StrLiteral StrNFind Spec StrToLabel StrWhileLabel'+
21. '|split'+
22. '|m:'+
23. '? shell ls ..\..\rrt\* -include *.html -name '+
24. '|c:fanout'+
25. '|chop .'+
26. '|l:'+
27. '|hby:nfind Hostby'+
28. '|str:nfind any /str/'+
29. '|nf:nfind NFind'+
30. '|f:'+
31. '? hby:'+
32. '|spec $<a href="Doc/GetHost.html">$ 1 1-* n $</a>$ n'+
33. '|f:'+
34. '? str:'+
35. '|if1:if all ^/Stru/ & ^/Strip/'+
36. '|if2:if find any /strnfind/'+
37. '|spec $<a href="Doc/Find.html">$ 1 1-* n $</a>$ n'+
38. '|if2:'+
39. '|spec $<a href="Doc/$ 1 4-* n $.html">$ n 1-* n $</a>$ n'+
40. '|if2:'+
41. '|if1:'+
42. '|f:'+
43. '? nf:'+
44. '|spec $<a href="Doc/Find.html">$ 1 1-* n $</a>$ n'+
45. '|f:'+
46. '? c:'+
47. '|spec $copy ..\..\rrt\$ 1 1-* n $Doc\$ nw'+
48. '|shell'+
49. '|cons'

What it does:

1. The shell stage executes a PowerShell command. In this case ls. It will list the names of all of the builtin stages, excluding those that are not stages.
2. Part of line 1 above.
3. Part of line 1 above.
4. The nlocate stage looks for the string File.cs. If the string is found it will discard that record. All other records are passed to the next stage.
5. The chop looks for a period in a record. If one is found it splits the record at that point. The portion of the record before the period is written to the next stage. The portion after the period is discarded.
6. Here we merge in stage name aliases and other stage names in order to maintain source campatibility with IBM. The faninany stage will read data from any connected stream when data becomes available on that stream.
7. Sort the stage names in alphbetical order. Since we do not supply any key ranges, the entire record is used as the sort key.
8. The lookup stage compares detail records against a set of master records. The detail records are read from the Primary input stream. The master records are read from the Secondary input stream, which is created on line 26. Detail records that have a matching master record are written to the Primary output stream. Those with no matching master record are written to the Secondary output stream, which is also created on line 26. Any record written to the Primary output stream has a documentation web page.
9. The spec stage is used to modify existing records and/or insert new records into the data stream. In this case we are creating a link to the documentaion page of the stage name that is on the current input record. We start by inserting <a href="Doc/GetHost.html"> into column 1 of the output record. Then we insert the contents of the current record on the Primary input stream into the next column of the output record. followed by the string </a>. This new record is then written to the Primary output.
10. Remember the lookup stage on line 8. The records it writes to is Secondary output, the ones with no matching master record, will end up here on the Secondary input stream. The faninany stage will read a record from any input stream that has data ready to be read.
11. On line 4 we discarded any record that contained the string "File.cs". Here we add them back in with a href to documentation that is known to exist.
12. See line 11.
13. See line 11.
14. This spec stage creates an HTML table cell.
15. The join stage will read a record and concatenates that record with the next 7 records. It then writes a record that contains 8 cell defintions.
16. This spec stage creates a table row from the 8 cell definitions.
17. By default literal stage writes the HTML table tag to its primary output before it reads and writes records from its Primary input stream. Thus the table tag will be the first record in the output file.
18. This literal stage has been directed to write its literal after it has read and written all of the data from its Primary input stream. Thus the closing table tag wiil be the last record in the output file.
19. The > stage, an alias for "CreateFile", opens the specified file and writes data from its Primary input stream to that file. If the file already exits it is over written otherwise a new file is created. This is the last stage of the first pipe in the pipeline.
20. This is the first stage of the second pipe. This literal stage inserts a string containing the aliases and other stage names eliminated above.
21. The split stage breaks a record into segements and writes each segment to its Primary output. In the case the break point is any space character.
22. This reference label m: sends the stage names to be merged in with the main data flow on line 6.
23. This is the first stage of the third pipe and it extractes the names of all of the documentation web pages.
24. The fanout stage writes a copy of each input reacord to all of its connected output streams. In the case the Primary and Secondary streams.
25. This stage will remove the file extension from the file name. See line 5.
26. Remember the lookup stage on line 8. This label reference creates the Secondary input and output streams for the lookup stage. The list of web pages goes to the Secondary input of the lookup stage. lookup reads the master records from the Secondary stream. If a detail record does not have a matching master record then it is written to the Secondary output of the lookup stage. The Secondray output goes to the stage following this one.
27. This stage looks for records that start with the string Hostby. If one is found it is written to the Secondary output stream.
28. This stage looks for records that begin with the string str. The compare is case insensitive. If a record is found it is written to the Secondary output.
29. This stage looks for records that beging with NFind. If one is found it is written to the Seconadry output.
30. This label reference creates the Secondary input to the faninany stage on line 10. This is the last stage of the third pipe.
31. This is the first stage of the fourth pipe. This label refernece defines the Secondary output of the nfind stage on line 27.
32. This stage sets the link tag to the GetHost page which is known to exits.
33. This label reference defines the Tertiary input to the faninany stage on line 10.
34. This is the first stage of the fifth pipe. This label reference str: defines the Secondary output of the nfind stage on line 28.
35. The if stage creates an If, Else and/or Endif construct within the pipeline. In this case the if defined by the label if1: is a If, Endif and the one defined by the label if2: is a If, Else, Endif. The if1: stage will load an all stage which looks for records that do not contain the indicated strings. If a record meets the condition specified by the all stage, the if stage will pass the record to its Primary output. If a record fails the condition set by the all stage it is written the to the Secondary output. The Primary output is the true path and the Secondary output is the false path thru the If, Else, Endif. All stages after the if stage upto the next occurence of the stream label if1: constitute the true path. All stages after the second stream label upto the next occurence of the label constitute the false path. When records on the true path reach the stream label they are witten to the Secondary input of the if stage. The if stage then writes its Secondary input to its Tertiary output stream, if connected. If the Tertiary output is not connected they are written to the Seconadry output. The Tertiary output is in effect the "Endif". When records on the false path reach the stream label they are written to the Tertiary input of the if stage. The if stage writes its Tertiary input to its Tertiary output.
36. This is the first stage of the true path for the if defined by the label if1:.
37. This is the first stage of the true path for the if defined by the label if2:.
38. The label refernce is the Else of the inner if.
39. This is the first stage of the false path of the if defined by the label if2:.
40. This label reference is the Endif of the inner if.
41. This label reference is the Endif of the outer if if1:.
42. This label reference defines the Quartinary input to the faninany stage on line 10.
43. This is the first stage of the sixth pipe. This label reference nf: defines the Secondary output of the nfind stage on line 29.
44. Create the documentation page link for the NFind stage.
45. This label reference defines the Quinary input to the faninany stage on line 10.
46. This is the first stage of the seventh pipe. It receives data from the fanout stage on line 24.
47. For each input file name this spec stage creates a command to copy that file to the documentation directory.
48. This shell stage executes the commands created by the previous stage.
49. The console stage writes any input the terminal screen. This lets the user now which files where copied.

To see the output produced by this pipe go here.
The run time for this pipe was 1.328125 seconds.

I take the table generated above and copy it into a web page. The following pipe takes that same table and converts it into a "index.html" web page for the "Doc" directory.

1. 'literal <HTML><Head>'+
2. '|literal append \<Title>Stage Documentation</Title>\'+
3. '|literal append \<META http-equiv="Content-Type" content="text/html; charset=UTF-8" />\'+
4. '|literal append \<META http-equiv="Content-Style-Type" content="text/css" />\'+
5. '|literal append \</Head><Body>\'+
6. '|append < stagetable.html'+
7. '|change \Doc/\\'+
8. '|literal append \</Body></HTML>\'+
9. '|> Doc\index.html'

1. Lines 1 thru 5 generate the web page heading. A note about the literal stage. With no oprions specified it writes its literal string and then reads and writes its Primary input. Thus without the append option the heading lines would be in the inverse order that they appear above. The append options says to read and write any input data and then write the literal string.
2. See line 1
3. See line 1
4. See line 1
5. See line 1
6. The append stage first reads and writes any input data. It then loads the specified stage and executes it. In this case it reads the generated table.
7. The generated table refers to the "Doc" directory. But the page will reside in the "Doc" directory. So we need to delete the "Doc" reference from the table. The change stage does this.
8. This literal stage add the closing tags for the web page.
9. Finally we write the "index.html" file.

Running Under PowerShell

To invoke a pipe from a PowerShell script you need to do the following:
1. Define the pipe as a string. For example:

$pipe = '(end ?) stem data'+
   '|f:find abc'+
   '|cons'+
 '? f:'+
   '|> notabc.txt'

2. To run the pipe use the Run-Pipe cmdlet provided with the application.
run-pipe $pipe

For more information on creating and using pipes under PowerShell see the HowTo.

Summary

This article serves as an introduction to the syntax used in the IBM manuals referenced below and in any documentation that I will supply. Also I hope shows the potential of PipeLines. Part II will be on data flow through a pipeline.

References

z/VM V5R3.0 Pipelines Reference
z/VM V5R2.0 CMS Pipelines User's Guide
CMS/TSO Pipelines Runtime Library Distribution
Author's Help Files

History