Process Pipelines

Process pipeline is a system to break down complex repeatable work into digestable steps that lead to concrete results.

Processes, unlike projects, don't have a defined completion time and goal. Their goal is to keep running and moving forward at a steady pace.

Examples of endless processes

• We brush teeth every day, there is no goal to that.

• We read new books once in a while, with no stop in sight.

• We continue learning to advance our goals and career throughout the life.

Kanban boards are great for visualising such pipelines, even for yourself. They work because:

• we can judge state of the entire process at a glance;

• "work in progress" and bottlenecks are visible;

• it is clear than one progresses by move things from one side to another

As engineers we have already intuition about long-running processes that break down work into multiple steps: data processing pipelines, sagas, actors, channels and routines, pipes, queues etc.

Process pipelines, when it comes to organisations and non-software things, are very similar in nature.

Instead of lambdas and code, steps are being executed by people. Frequently by ourselves, in a different mood and context.

We could use that similarity to design and run such pipelines.

For example, got too many cards in middle columns? Something is wrong with the process, bottleneck will just keep on growing and consuming RAM. We need to apply back-pressure or reduce the load on processors.

Work flow

Everything looks right, but work just doesn't flow? Let's measure queue processing latency at every step, identify the biggest factor eating at our time budget and start by optimising it.

For example, when wearing "Manager Hat" (will be covered in “Two Hats” blog post) I use Kanban to break down work and prepare it for my worker persona:

• design the structure to maximise the flow;

• Deep linking cards to actual items to reduce context switching;

• let worker pick any card, as long as WIP stays minimal and process runs.

This approach is also at the heart of my writing process. It works quite good there.