Click here to Skip to main content
Click here to Skip to main content

CMS/TSO PipeLines for Windows - Part III User Stages

, 29 Aug 2008
Rate this:
Please Sign up or sign in to vote.
An implementation of IBM's PipeLines for Windows and PowerShell
Download PipeLib Class Library

Introduction

CMS/TSO PipeLines is a product that runs on IBM mainframes. On a z/VM system it runs uner CMS. On a z/OS system it runs under TSO. As the name suggests data is passed thru a series of builtin or user defined stages. Along the way the data can be modified or deleted from the pipe. You may be saying to yourself there is nothing new in that. What is different is that PipeLines supports concurrent parallel pipes and data can be routed into and out of a pipe.

System Requirements

While the PipeLib class library can be accessed from a .Net program, I think you will find that running it under PowerShell more useful. I do all of my testing through PowerShell and any pipelines I supply will most likely be PowerShell scripts. Therefor as minimum system requirement's I recommend the following.
PowerShell V1.0
.Net Framework V2

Invoking a User Stage

A user stage is invoked by the PS built in stage. This stage takes as parameters the name of the user written stage and any parameters the user wishes to pass to the user stage. The basic outline of a user stage is as follows:

#BasicStage.ps1

1)param ([object] $usp, [String] $parms)

2)  ... Perform any needed setup, such as parsing the passed parameter string ...

3)  usp-peekto 'inputRec'
4)  while ($rc -eq 0) {

5)    ... process the record

6)    usp-output 'outputRec' 
      if ($rc -eq 0) {
7)      usp-ReadTo
8)      usp-PeekTo 'inputRec'
      }
    }
Line 1 is the only line that is mandatory. It must be present in every script that uses any of the user stage commands. Lines 3-8 sets up the basic data flow needed to keep the data records in sequence. This is not enforced and the user is free code what ever is best for the application.

The $usp variable defined in line 1 is the interface that connects the script to the PipeLib class library. The pipe user commands check for the presence and the validity of this variable.

Pipe User Commands

Below is the list of commands available in a pipe user stage.

Run-CallPipe
This command executes a pipeline in parallel with the current pipeline. The user stage is suspended while the new pipeline runs. The user stage resumes when the CallPipe command ends. The string that defines the new pipeline is passed as an argument to this command.
Examples: run-callpipe $pipe or run-callpipe "pipe definition"
Run-AddPipe
This command executes a pipeline in parallel with the current pipeline. Unlike the CallPipe command the user stage continues to run. The string that defines the pipeline is passed as an argument to this command.
Examples: run-addpipe $pipe or run-addpipe "pipe definition"
Usp-PeekTo
This command reads without consuming the next record passed to the stage. The name of a variable to receive the record is passed as an argument.
Examples: usp-peekto "varname"
Usp-ReadTo
This command reads and consumes the next record passed to the stage. The name of a variable to receive the record is an optional argument.
Examples: usp-readto "varname" or usp-readto
Usp-Output
This command writes data to the currently selected output stream. The output data is passed in a variable or a literal string.
Examples: usp-output $line or usp-output "literal data string"
Usp-Select
This command selects the data stream to used for input and/or output. The primary input and output are selected by default when a user stage is started. This command requires two parameters:
  1. The type of stream to process: "Input" or "Output"
  2. Stream number or identifier.

Examples: usp-select "both" 0, usp-select "input" 1, usp-select "input" "stream-name"
Usp-Commit
This command sets the commit level this stage is to be set to. The new commit level is a required parameter.
Examples: usp-commit 0
Usp-NoCommit
This command prevents the user stage from auto committing.
Examples: usp-nocommit
Usp-ScanRange
This command scans a data string for a location range parameter. The command takes four parameters:
  1. Indicates if the location parameter is required or optional and assumes one of the following values: Required or Optional
  2. A variable name to receive the generated range location token.
  3. The name of a variable to receive the remainder of the data string after the location has been parsed off.
  4. The fourth parameter is optional. It specifies data to be parsed. If omitted the parameter string passed to the user stage is used.

Examples: usp-scanrange "required" "token" "rest" "fs - f1", usp-scanrange "optional" "token" "."
Usp-GetRange
This command extracts the data specified by a previous ScanRange from a data record. The command takes four parameters:
  1. The token generated by a previous call to ScanRange
  2. The type of variable to receive the extracted data. It can assume one of the following values: "VARiable" or "STEM".
  3. The name of a variable to recieve the extracted data.
  4. The data record from which the data is to be extracted.

Examples: usp-getrange $token "var" "varname" $data, usp-getrange $token "stem" "stemname" $data
Usp-ScanString
This extracts the next word or delimited string from a data string. The command takes three parameters:
  1. The name of the variable to receive the string.
  2. The name of a variable to receive any residual data.
  3. The data string from which the string is to be extracted. This parameter is optional. If omitted the parameter string passed to the userstage is used.

Examples: usp-scanstring "string" "rest" $data, usp-scanstring "string" "."
Usp-Short
This command shorts the currently selected input stream to the currently selected output stream.
Examples: usp-short
Usp-Sever
This command severs the currently selected input or output stream. The single parameter specifies which stream to sever. If the severed stream was redirected by a CallPipe or AddPipe connector, Then an attempt is made to restore the original connection.
Examples: usp-sever "input" or usp-sever "output"
Usp-StreamNum
This command gets the stream number of the specified stream. The command takes two parameters:
  1. The type of stream to process: "Input" or "Output"
  2. Stream number or identifier. This parameter is optional. If omitted the currently slected input or output stream is used.
If the stream is connected the $rc variable will contain the stream number. If not connected it will contain a -4.
Examples: usp-streamnum "input" 1, usp-streamnum "output" "id"
Usp-StreamState
This command queries the status of the specified stream. The command takes one or two parameters:
  • When the first parameter is "Input" or "Output", the second parameter, if specified, is the number, or id, of the stream to query. The default is the currently selected stream.
  • When the first parameter is "Summary", a return code of 0 indicates that at least one input and one output stream are connected.
  • When the first parameter is "All" then the second parameter is the name of variable to recieve the status of each pair of streams.

Examples: usp-streamstate "input" 1, usp-streamstate "all" "varname"
Usp-MaxStream
This command returns the max stream number for the selected stream type The command takes one parameters:
  1. The type of stream to process: "Input" or "Output"

Examples: usp-maxstream "input", usp-maxstream "output"
Usp-AddStream
This command adds an unconnected stream to the user stage. The command takes two parameters:
  1. The type of stream to process: "Input", "Output", or "Both"
  2. Stream identifier. This parameter is optional. If omitted then use Usp-MaxStream to get the number of the added stream.

Examples: usp-addstream "input" "strmid", usp-addstream "both"

AddPipe and CallPipe

Of the above commands CallPipe and AddPipe require special attention. These commands insert a parallel pipeline into the existing pipe line and depending on the options coded for parallel pipe it may or may not interact with the existing pipeline. The pipe definition syntax for the CallPipe and AddPipe commands contain one additional element over that of the Run-Pipe command.

                                                   ┌─<─endchar┐
>>─┬AddPipe─┬┬─────────────────────────┬┬─────────┬┴┤ PipeLine├┴─><
   └CallPipe┘│  ┌─<─────────────────┐  │├stagesep─┤
             └(─┴┬┬STAGESEP──┬xorc─┬┴)─┘└endchar──┘
                 │└SEPerator─┘     │
                 ├ENDchar xorc─────┤
                 ├ESCape xorc──────┤
                 ├NAME─pipename────┤
                 └TRACE────────────┘

Group PipeLine

               
├─┬───────────┬─>
  └┤connector├┘ 
                

  ┌─<─stagesep───────────────────────────────────────────────┐
>─┴┬────────────────────────────────────────────────────────┬┴─>
   ├┬────────────┬┬───────────────┬stagename─┬─────────────┬┤ 
   │└┤LabelGroup├┘└(─┬───┬TRACE─)─┘          └stageoptions─┘│
   │                 └NO─┘                                  │
   └┤LabelGroup├────────────────────────────────────────────┘

>─┬───────────┬─┤
  └┤connector├┘


Group <a id="""Connector""">Connector</a>

├─*─┬───────────────────────────┬:──┤
    │           ┌.*────────────┐│
    └.─┬INPUT──┬┼──────────────┼┘
       └OUTPUT─┘└.─┬*─────────┬┘
                   ├streamnum─┤
                   └streamid──┘

Group LabelGroup

├─┬────────────────┬─┤
  ├label:──────────┤
  └label.streamid:─┘

  • Options
  • STAGESEP xorc
    SEPerator
    Specify the character to be used to separate the stages of a pipe. It can be specified as a single character or as the hex digits of a character. The default separator is the virtical bar, "|".
    ENDchar xorc
    Specifiy the character that separates the pipes of a pipeline. It can be specified as a single character or as the hex digits of a character. There is no default.
    ESCape xorc
    Define an escape character that can be used to override the processing of a character that has special meaning to the PipeLine parser. Any character that follows the escape character is treated as an ordinary character. There is no default.
    NAME pipename
    Specify a name to be associated with this pipeline.
    TRACE
    Turn on tracing for the pipeline.
  • Pipeline
  • stagesep
    The stages of a pipe are separated by the STAGESEP character. The default is "|".
    endchar
    If there are multiply pipes in the pipeline, separate them by the ENDCHAR.
    NO
    If tracing is turned on for the pipeline then "NO TRACE" will turn it off for this stage of the pipeline.
    TRACE
    Turn tracing on for this stage of a pipeline. If prefixed by the "NO" keyword then tracing is turned off for this stage.
    stagename
    The name of the stage to be executed.
    stageoptions
    Supply any options needed by the specified stagename.
    *
    Identifies this as the start of a connector definition.
    INPUT
    This connector applies to an input stream of the user stage. If the connector is the first stage of a pipe then the user stages input stream is directed to the first stage of the CallPipe or AddPipe pipeline. If the connector is the last stage of a pipe then the output of the CallPipe or AddPipe pipeline is directed to the user stage input.
    OUTPUT
    This connector applies to an output stream of the user stage. If the connector is the first stage of a pipe then the user stages output stream is directed to the first stage of the CallPipe or AddPipe pipeline. If the connector is the last stage of a pipe then the output of the CallPipe or AddPipe pipeline is directed to the user stage output.
    *
    This connector applies to the currently selected input or output stream.
    streamnum
    The stream number of the effect user stage data stream.
    The stream ID of the effected user stage data stream.
    :
    Marks the end of a connector definition.
    label:
    label.streamid:
    A label that identifies additional input and/or output data streams for the stage it is specified on. The first occurrence of a label is called a label definition. It establishes connection point where other stages, in other pipes, receive data from or send data to. Each subsequent use of the same label is called a label reference. Use label references to define additional input and output streams for the stage. To use a label reference, specify a stage containing only a label with no stagename.

    A streamid assigns a symbolic name to a stream. Name a stream by adding an identifier to the label. Write the label immediately followed by a period (.) and up to 4 alphabetic characters or a combination of alphabetic characters and digits that includes at least one alphabetic character. A stream identifier must be immediately followed by a colon with no intervening blanks.

The Connector element is used to redirect the data streams of the user stage to the pipeline defined by the CallPipe or AddPipe.

Consider the pipe "A|Z|D" where "Z" is a user stage that issues a CallPipe or AddPipe. The following demostrates the effect of connectors on data flow through a pipeline when CallPipe is used.

CallPipe:    B|C      *.Input:|B|C   B|C|*.Output:    *.Input:|B|C|*.Output:

              A            A           A                   A    
              │ B          └─B         │ B                 └─B
DataFlow:     Z │          Z │         Z │                 Z │
              │ C          │ C           C                   C
              D            D           D─┘                 D─┘ 

Restorable:  NA            Yes        Yes                 Yes
             (1)          (2)         (3)                 (4) 
  1. With no connectors, data flows from one stage to the next in each pipeline.
  2. A's output stream is directed to B's input. Z's input stream is disconnected.
  3. C's output stream is directed to D's input. Z's output stream is disconnected.
  4. A's output stream is directed to B's input. C's output stream is directed to D's input. Z's input and output streams are disconnected

The following demostrates the effect of connectors on data flow through a pipeline when AddPipe is used.

AddPipe:    B|C   *.Input:|B|C  B|C|*.Output:  *.Output:|B|C  B|C|*.Input:
                                                            
             A        A           A               A             A  B       
             │ B      └─B         │ B             │                │       
DataFlow:    Z │      Z │         Z │             Z             Z─ C       
             │ C      │ C         ┌─C             └─B           │         
             D        D           D               D │           D         
                                                    C 
Restorable:  NA      No          No              Yes           Yes                                                   C                     
            (1)      (2)         (3)             (4)           (5)        

AddPipe: *.Input:|B|C|*.Output:   *.Input:|B|C|*.Input:

               A                       A  
               └─B                     └─B
DataFlow:      Z │                       │
                 C                       C
               D─┘                     Z─┘
                                       │  
                                       D

Restorable:    No                      No
              (6)                      (7)

AddPipe: *.Output:|B|C|*.Input:   *.Output:|B|C|*.Output:

               A                       A  
             ┌─┐ ┌─B                   │ ┌─B
DataFlow:    │ Z │ │                   Z │ │
             │ └─┘ C                   └─┘ C
             └─────┘                   ┌───┘  
               D                       D  
                                       
Restorable:   Yes                      No
              (8)                     (9)

  1. With no connectors, data flows from one stage to the next in each pipeline.
  2. A's output stream is directed to B's input. Z's input stream is disconnected.
  3. C's output stream is directed to D's input. Z's output stream is disconnected.
  4. Z's output stream is connected to B's input. D's input stream is disconnected.
  5. C's output stream is connected to Z's input. A's output is disconnected.
  6. A's output stream is directed to B's input. C's output stream is directed to D's input. Z's input and output streams are disconnected
  7. A's output stream is connected to B's input. C's output stream is connected to Z's input.
  8. Z's output stream is connected to B's input. C's output stream is connected to Z's input. A's output and D's input streams are disconnected. All stall is possible.
  9. Z's output stream is connected to B's input. C's output stream is connected to D's input.

An Example

Below is an example of a pipe that invokes a user stage.

#AddPipeDemo

$data = "B999 1323 FFFF BA82 1A43 20DD"

$pipe =    'var data'+
  '|split'+
  '|PS userstage'+
  '|stem Stem1'

run-pipe $pipe
The user stage called by the above pipeline is:
# AddPipeDemo userstage

1) param ([object] $usp, [String] $parms)

   # Read parameter file
2) run-addpipe '< addpipe.conf | *.input:'     # Connect input to file       
3) usp-nocommit                                # Disable automatic commit    
4) usp-readto 'line'                           # Read first line of file     
   while ($rc -eq 0) {                         # Process all lines           
     if ($line[0] -ne '#') {                   # Ignore any comments
            usp-scanstring 'sortOrder' '.' $line
           write-host "The selected sort order is: $sortOrder"
     }
     usp-readto 'line'                         # Read next line              
   }
   $order = 'fa-ff'
   if ($sortOrder -eq 'Lower') {
       $order = '0a-0f'
   }
5) usp-sever 'input'                           # Re-instate input file       
6) usp-commit 0                              

7) $pipe =    '*.input:' +         # connect to output of stage preceding caller
     '|xlate 1-* A-F '+$order+   # Translate A to F 
     '|sort' +                   # Sort the records
     '|xlate 1-* '+$order+' A-F'+ # Restore original characters A to F
     '|*.output:'                # connect to input of stage following caller

8) run-callpipe $pipe
  1. The mandatory parameter definitions requied by a user stage.
  2. Initiate the pararell pipe. The output of this pipe is directed to the primary input of this stage. The previous input stream is svaed for a posible reconnection later.
  3. This stops the user stage from doing an Auto Commit. All stages start at some commit level. For user stages it is -1. Most of the builtin stages start at some level, verify the state of the stage and then commit to level 0. Stages at the lowest commit level run before stages at a higher commit level. All builtin stages will eventually end up at commit level 0. Thus with this command this stage will stay at level -1 until it explicitly commits to a higher level. Normally any IO request by a user stage causes an Auto Commit to level 0.
  4. Read the first input record. Note that this read will consume the record. There is no need to worry about record sequence because the records never leave this stage.
  5. After all records have been processed we restore the original input stream.
  6. Here we commit to level 0. The other stages will have already commited to level 0 and are waiting for this stage to join them. Once all stages are at level 0 they are all eligible to run.
  7. Define a pipeline to be executed.
  8. Execute the pipeline.

Another Example

The following example of a user stage takes records that conatin binary data and dumps them in hex format to the screen.

#Dump.ps1

param ([object] $usp, [string] $parms)

usp-scanstring 'width' '.'
if ($width -eq '16') {
  $text = ' 1-* 45 '
} else {
    $width = '32'
  $text = ' 1-16 83 17-* 101 '
}    

$pipe = '(end ?) *.Input:'+
  '|deblock fixed '+$width+
  '|f:fanout'+
  '|copy'+
  '|s:spec set (#1=1; #2=8; #3=0)'+
  '  while (#1<length() && #3<4) do'+
  '    print (c2x(substr(record(), #1, 4))) (#2, 8)'+
  '    set (#1+=4; #3+=1; #2+=9)'+
  '  done'+ 
  '  set (#2=45; #3=0)'+
  '  while (#1<length() && #3<4) do'+
  '    print (c2x(substr(record(), #1, 4))) (#2, 8)'+
  '    set (#1+=4; #3+=1; #2+=9)'+
  '  done'+ 
  '  select 1 '+$text+
  '  pad 0 print (d2x(#0)) 1.4 r'+
  '  set (#0+='+$width+')'+
  '|*.output:'+
  '? f:'+
  '|xlate 00-1f . 80-9f .'+
  '|s:'
  
run-callpipe $pipe  
An sample of the output produced by this stage, called with a width parameter of 16:
0000   3f523f4e 54465320 20202000 02080000  ?R?NTFS    .....
0010   00000000 003f0000 3f003f00 043f0000  .....?..?.?..?..
0020   00000000 3f003f00 11583f03 00000000  ....?.?..X?.....
0030   3f7f0c00 00000000 3f024000 00000000  ?⌂......?.@.....

References

z/VM V5R3.0 Pipelines Reference
z/VM V5R2.0 CMS Pipelines User's Guide
CMS/TSO Pipelines Runtime Library Distribution
Author's Help Files

License

This article, along with any associated source code and files, is licensed under The GNU General Public License (GPLv3)

Share

About the Author

WBurgMo

United States United States
No Biography provided

Comments and Discussions

 
QuestionStatus at this point, and question on TCPCLIENT Pinmemberjohn bacas20-Aug-09 17:04 
AnswerRe: Status at this point, and question on TCPCLIENT PinmemberWBurgMo21-Aug-09 9:18 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140814.1 | Last Updated 29 Aug 2008
Article Copyright 2008 by WBurgMo
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid