Workflow Basics - Getting the Process Right

Introduction

This article isn't actually specific to WWF, but it does provide some tips for designing workflows that work well. This article isn't written to follow the specific meanings that WWF and other workflow models use for certain words, so take things with a grain of salt when reading.

Background

Many years ago (and for several years), I did some design work for a UI process flow engine and some very complex workflows that ran within it. Most of these workflows had a few hundred pages of documentation. Along the way of actually designing the workflows, a number of simple rules quickly emerged about how to do things better. These rules principally relate to workflows that are used to gather information from users (or others, e.g., other businesses that interact with you through your workflow), but most of them will have general applicability.

The workflow rules (learnt the hard way)

1. Do we have enough buckets for all of the leaks?

When your workflows are complex, you can end up in a situation where you have a peculiar combination of fringe cases that you didn't plan for. These fringe cases can end up 'leaking' out of your workflow and being lost, although in reality, it's more like they end up 'pooling' somewhere unobserved - probably causing rot to set in. It's a good idea to ensure that every path goes somewhere and actually exits, even if you have to empty them into a 'bucket'.

For instance, a particular instance of a workflow may go 'on hold' forever waiting for some user interaction that's never going to come because the user in question quit and went to work elsewhere. In general, you should consider everything as having the chance of getting backed up (in a blocked drain sense), especially those that will *never* get blocked, and plan for it accordingly. Handling timeouts anywhere and everywhere is a particular type of bucket that you should always have in place.

2. Kill them quickly

One simple way to optimise workflows is to explicitly test for all of the conditions that would lead to a workflow instance being abandoned and put these very early in the workflow definition. The idea is to avoid performing steps that ultimately aren't going to be necessary.

3. Don't leave 'em hanging

Where you are gathering a lot of information from someone (not necessarily your user, but perhaps your customer), there can often be a desire to just throw a free-form field at them and let them get on with it. This is especially the case when the goal is to facilitate as many chances as possible for the person to give a 'complete' set of information or because the person is not considered 'trustworthy'. Sometimes there's a desire to try and 'catch out' the person supplying the information, i.e., "give them enough rope and they'll hang themselves". So one or more free-form text declarations are thrown at them that will later require human interpretation.

This approach is genuinely not conducive to converting to a successful workflow. In fact, from a business perspective, such an approach is almost guaranteed to fail in its objectives regardless of how you implement it (even with paper forms). The real answer is to carefully analyse for all of the possible options, adding a 'bucket' at each step for anything you may not have thought of, and carefully guiding the person in question to make a full and honest declaration. As you review what falls into the 'buckets', you can refactor your workflow and improve it incrementally.

4. Ask and ye shall receive

That point about avoiding unnecessary steps is also key to keeping on top of your workflow design in general. It's important to evaluate each step in the workflow and see whether it's really necessary at that particular point. If not, move it down; there's no point in performing certain actions or requesting certain information simply because that's the way you always do it. That's just being wasteful.

If your workflow is doing something like gathering the same piece of information or asking the same person for approval multiple times (think declaration forms), then stop it. Ask once, no more. Many paper based workflows have a form pass under someone's nose for initial approval and then back under their nose for verification and sign off. You don't need to do this. Redesign your workflow so that it only goes back past someone if something material has been changed.

5. See the wood and the trees

When writing code, we often want to target the primary functionality and eliminate as much as possible of anything that's secondary, at least until a later release. This is also true in workflows, but there are important considerations. The whole idea of a workflow is often to capture and express business knowledge and domain experience with regards to how a particular task is done, but if we only stay at doing 80% of the job, we're missing the point of putting it in a workflow in the first place.

For example, we could have a process that has too many 'buckets'. Too many cases where we're abdicating from the workflow dealing with something automatically, and instead we're going back to that domain experience that we're supposed to be capturing to fill in all of these gaps manually. We're not really leveraging the strength of the workflow if we stop at 80%.

To illustrate, imagine that you've implemented a workflow that covers 99.99% of all the different cases by volume. That's great, only 1 in 10000 requires intervention, and you only process that many in one week! It would be great if the intervention only required a minute or two, but generally, the fringe cases require a lot longer than that to resolve, more like a few hours to a few days every time. That's a massive amount of wasted time.

Now if you start to get a pattern together about how you deal with these fringe cases so that they don't take so long, isn't it about time that you put that into the workflow and capture that rare, but very valuable, business knowledge?

6. Yes, the tail does wag the dog

As you go about adding the fringe cases, you'll find that your workflow definition will probably require some pretty spectacular refactoring to fit them all in - this is normal. It wouldn't be uncommon to see seemingly trivial issues elevated high up the workflow because they are in fact highly discriminatory (although they may still seem unimportant), or because putting them there helps you shorten a particular path through the workflow.

7. Abstract, abstract, abstract

Always be on the lookout for any patterns that emerge, and try to develop workflows that fit those abstract patterns. This will really open your mind to the process that you're trying to describe, and truly enable you to put things in the right place in the workflow. Plus, it will help cut down the documentation and testing.

8. Data Structures <> Business Objects <> Workflow <> User Interface

When it comes to actually gathering information from a user through a workflow, there are some things that are definite no-nos. We could rename this section 'unwanted exposure' to give you a clue what these no-nos are.

One of the biggest mistakes is to take your business data and stick it unadorned in front of your user, or organise your workflow so that it won't get messed up by referential integrity issues in your schema. This is exposing some of the deepest darkest secrets of your business to the outside world. Not to mention the point that if what your user is entering is too close to your database, then you may have bigger potential issues with SQL insertion attacks.

Surely business objects must be the answer then? The answer to that would be no also.

Your workflow is a process, not an object.

Apologies for stating the obvious, but all too often this is forgotten. For instance, a mistake, almost as large as the one above, is to assume that your business objects will make perfect sense to your user, or that if they don't, then extensive training will help sort that out.

Trying to get your user to 'fill-up' your business object or database table, because that's the information you have to get, isn't just wrong - it's stupid. It's exposing the secrets of your business, when all you're supposed to be exposing is an interface to your business. If you have to train users to be able to interpret the meanings of all of the different fields that they see, then you've exposed the way you do your business to them - what if your user is actually a competitor of yours?

Your business objects should support the workflow - not dictate it.

Instead, what you show to your user in the user interface should make sense to them, and that's highly unlikely to be the same thing that you store in your database or business objects. You'll also want the answers that the user gives you to guide the workflow so that the sequence of activities they are asked to perform seem logical to them. This is another way in which abstraction comes to the fore (in a different sense). The database and business objects are your domain, don't go exposing it to the user needlessly. Instead, abstract away the specifics of your system so that you can ask what you need to ask, when you need to ask it.