New Flow State Machine pattern (with unlimited flow duration & easy to setup) : the “Flow Controller”

Last year (in april 2017) I mentioned that I was a little bit worried that Microsoft Flow didn’t provide state machines workflows; I also mentioned that an easy way to achieve this was a loop and a goto variable. After testing that in real cases, I did publish a more detailed blog post on it.

In October 2017 I illustrated how to escalate approvals in flow.

In November 2017 I illustrated how to have approval analytics (including escalation) with Flow and PowerBI.

In January 2018, Michael Svenson illustrated how to achieve more than 30 days by splitting flows in serviced flows that recall themselves.

in 2018, I have tried to improve al lot of things:

• the state machine pattern (this post)
• the approval analytics (to be published soon)
• (this is very new) : how to minimize approvals by using Machine Learning (see my next blog post). (to be published soon)

The old (& loop based) state machine patterns works well, but it doesn’t not handle 1 basic issue : flows and flow approval stop after 30 days. So in my old state machine implementations I used to have a parallel branch that checked the number of running days and that sent a warning message to a supervisor (or to the approvers) after less than 30 days (usually 25 days). So far so good. But some customers want…unlimited time.

In this blog post I will illustrate an improved version of the state machine that fixes all these issues.

In my opinion approval workflows must at least:

• support state machines (or be able go back and forward to any other states)
• run more than 30 days (not out of the box in flow)
• support approval escalation (this is easy in flow)

Of course everything should be easy to implement and to maintain.

When I wrote my state machine blog post in November, I thought that the flows will be soon able to run for more than 30 days because this issue will eventually be fixed. At the time of writing this is not fixed yet. But in the meantime we need to live with that.

How can we combine all the requirements defined above ?

Obviously we need to split the approvals sections in approvals flow and after max 29 days the flows must be restarted.

But I wanted a solution that scales physically (> 30 days) and logically : by “logically”, I mean that I didn’t want an approval flow to call another approval flow directly, I wanted low coupling (we should not have to go to every sub flow to change the whole flow logic), but I also wanted to be able to proceed to approval escalations. So how can we combine the benefits of my old state machine workflow pattern with 1 branch for each approval level and approval escalations, but also > 30 days ?

Another issue with our previous loop based pattern was that debugging a live running loop is a nightmare in flow : we can’t. We can only debug a postmortem loop.

Welcome to the State Machine Controller pattern

Let’s take the following business case (see next picture):

a user publishes a document in the intranet

• Level 1 : the document will be approved by a group of users called “Advisers”
• Level 2 : if approved the document will be validated by a group of users called Advisors
• Level 3 : if approved the document will be validated by a group of users called “secretariat”
• Level 4 : if approved the document will be validated by the Big boss and the document can be visible to the public

…if rejected at any level the document approval will go back to the previous level

We already did implement this in the past with our (loop based) state machine pattern.

Each level can potentially need more than 30 days to complete, but that is the easy part,: we split the flow in several sub approval flows and after 29 days we restart the sub approval flow : there is no other technical solution, no discussion here.

But we wanted the flow logic to be centralized as we did in our old state machine pattern: each approval sub flow does not decide what is the next flow : a controller is supposed to orchestrate that; so each approval flow is just supposed to provide some information to the controller.

Architecture overview

Implementation

There are different ways to implement this pattern. Here I will illustrate just one implementation that I find very simple (other implemention could be based on message queues, service bus,…)

Let’s take a look a the Level 2 approval flow (level 2 is more interesting because we can go backward to level 1 and forward to level 3)

The flow is a service flow, so it starts with When a HTTP Request is received action.

If we reach 29 days, the approval is in timeout (we define 29 days as the timeout in the Start an approval action settings) and the error handler will run the action rerun the flow : this means that the approval will be removed and asking for restarting the flow will invoke the flow controller with the following parameters :

• “From” : “level2”
• “response”:”timeout” (here “response” could also be “escalate” if we want to escalate the approval)

If the user approves before 29 days, the flow will run the controller and pass the following values :

• “From” : “level2”
• “response”:”<the approval output : approve or reject”

Let’s take a look at the Controller implementation

Here is a high level view of the controller flow

Basically we have 1 branch for each approval (like our previous pattern with the loop), but there is no loop anymore because a loop means a state based architecture and we need it to be stateless because the code is supposed to be re-entrant.

The controller must be a service flow because it must be invoked by the main flow and also by approval flows. This is the reason why the trigger is When a HTTP request is received

The controller can receive 2 parameters :

• from : this is the name of the approval flow calling the controller (level1, level2, level3…)
• result : this is the approval flow result : “rejected”, “approved”, “timeout”, “escalate”

This is followed by a switch that will check the “from” value : if the “from” is empty then flow will call Level 1 Approval flow by default because we start the approval somewhere.

Let’s take a look at what will happen if the “from” returns “level2”:

for each branch (level1, level2, level3,…), there is another internal switch that checks the approval flow “result” value.

So for instance if the result value of level2 is “rejected” then the Level 1 approval flow will be started; if it is “approved”, the Level 3 approval flow will be invoked and so on…

If a result value is escalate, then then an escalating flow will be invoked.

We can make this architecture more generic by passing the approvers names, the approver message, and other options as parameters if needed.

But again the point is: don’t add to much flow logic in the approval sub-flows, but centralize it in the approval controller, and just pass messages to the controller. Each approvals sub-flow are not supposed to know each others, low coupling is key.

Now I’m aware that this architecture eats your flow quota, but flows are very cheap (you can buy 50.000 additional flow runs for 40$). I’m also aware that logging (and history) is split in several flows. To be honest, I implement my own custom approval history in rder to provide some complex flow approval analytics, so my approval information is stored in a custom SharePoint list or in a SQL Azure database.

Also this architecture is easy to implement and to maintain : of course I wish we had a state machine designer in flow and unlimited runtime, but this should not be a showstopper.

Just for the fun part : we can consider that a logical flow is split in several physical flows. I have a name for these logical flows :

cross-flows 🙂

In summary, the controller architecture is a nice way to implement complex state machine approval flows that take more than 30 days to complete, with approval escalation and centralized management and low coupling.

Happy flow’ing !

 

Update March 2019 ! I published an implementation of the flow controller pattern that doe not use the P1 Http action. Read this blog post.