Implementing waiting for approval in Prefect 2

In certain scenarios, you may want to continue processing your flow only once you’ve checked the status of previous processing.

Concrete scenarios

This typically involves some semi-manual processes that involve both some automation and some human component, e.g.:

  1. Run automated data extraction, transformation, and processing; then automatically inform some stakeholders about the status and let them know this data/process is ready for a manual quality check; once this person manually gives their approval, the flow run can continue.

  2. Run automated data preprocessing and feature extraction and send an alert to a data scientist that this process has been completed; once this person approves, the flow run can continue with downstream tasks that may, e.g., train an ML model with that preprocessed and manually validated data and extracted features.


Implementation

We plan to support pausing and resuming workflows, as well as tasks that require manual approval as a first-class feature. You will be able to specify that as part of your workflow definition.

:bulb: If you’re curious how this will (likely) work - Prefect will check an orchestration rule indicating whether the flow or task run is allowed to enter a Running state - 1) if not, it will wait for a certain amount of time and set the state to e.g. Paused/AwaitingApproval (just an example), 2) if yes, it will start running.


Temporary workaround solution

:warning: Note that the example below is just a temporary workaround until we have a first-class way to handle this type of use case.

First, create a JSON block.

Create a Block to store information for each process requiring approval

You can create a block either from the UI or from code, e.g., using code as shown here:

from prefect.blocks.system import JSON

block = JSON(
    value=dict(
        process_1=True,
        process_2=False,
        process_3=False,
        process_4=False,
        process_5=True,
    )
)
block.save("approval", overwrite=True)

Write your flow with business logic that requires manual approval

Here, we show a very basic flow just to demonstrate the pattern - adjust it to your needs.

The flow below does the following:

  1. Starts some initial processing
  2. Sends a Slack alert notifying about its completion with a link to approve the status by editing the approval flag on the Block
  3. Runs a while loop that polls for the current state of the approval flag set on the Block. The loop ends as soon as the approval flag changes to True.
  4. Once approved, continues processing of some critical part of the process that could run without prior approval.
  5. Finally, the flow sets the approval flag back to False so that subsequent runs can start from a clean slate.
from prefect import task, flow
from prefect import get_run_logger
from prefect.blocks.system import JSON
from prefect.blocks.notifications import SlackWebhook
import time


def send_alert(message: str):
    slack_webhook_block = SlackWebhook.load("hq")
    slack_webhook_block.notify(message)


def process_approved(process: str, block_name: str):
    block_value = JSON.load(block_name).value
    return block_value[process]


def set_approval_flag_back_to_false(process: str, block_name: str):
    block_value = JSON.load(block_name).value
    block_value[process] = False
    block = JSON(value=block_value)
    block.save(block_name, overwrite=True)


@task
def run_initial_processing():
    logger = get_run_logger()
    logger.info("Processing something important 🤖")
    logger.info("Calculating the answer to life, the universe, and everything...")


@task
def run_something_critical():
    logger = get_run_logger()
    logger.info("Got approval to reveal the answer to life, the universe, and everything!")
    logger.info("The answer is... 42!")


@flow
def semi_manual_process(
    process: str = "process_1",
    block_name: str = "approval",
    poll_for_approval_every_sec: int = 5,
) -> None:
    logger = get_run_logger()
    run_initial_processing()
    url = "https://app.prefect.cloud/account/c5276cbb-62a2-4501-b64a-74d3d900d781/workspace/aaeffa0e-13fa-460e-a1f9-79b53c05ab36/block/2c384be5-2975-450a-b9a5-7e126f9def37/edit"
    send_alert(f"{process} finished. Please approve to continue processing: {url}")
    while True:
        if process_approved(process, block_name):
            logger.info("Process got approved! 🎉 Moving on to the next task")
            break
        else:
            logger.info("Waiting for approval...")
            time.sleep(poll_for_approval_every_sec)
    run_something_critical()  # post-processing, ML training process, reporting on KPIs
    set_approval_flag_back_to_false(process, block_name)


if __name__ == "__main__":
    semi_manual_process("process_2")

Got it. I will wait until it is released as a feature and I am happy to here that it will be supported similar to 1.0. While I see how this could work (we did something similar with KV store in 1.0 before), it is ultimately pretty inefficient for when you have long periods of pausing. You would be paying for compute for the duration of the pause until the block was updated. In 1.0 running on ECS Fargate the ECS task shuts down and deprovisions and then a new ECS task runs when unpaused… this means you aren’t using any compute during long periods of pause. In our use case we have several instances where things pause for several hours.

1 Like

In that case, it’s probably better not to wait but instead leverage the orchestrator pattern in combination with e.g. blocks (similarly to how this is shown here but using separate flow run processes rather than tasks):

I’m not sure I completely follow the orchestrator pattern concept. Is there more docs or examples I can reference.

I am not sure this is going to be a solution for us. While I appreciate that it might work from a technical level I think from a business use case it will create too much noise. We use this pause/resume flow structure for hundreds of flows and it provides the ability for us to encapsulate those complete pipeline processes that involve multiple outside services to a single view within Prefect. When we break down into smaller pieces that would be multiple flows and runs we lose that ability to have a complete view of a pipeline in a single place. Given the number of flows involved this sounds like it could be increasingly messy and hard to manage.

I get you, we need more examples on that for sure, let’s see how we can prioritize that

for now, definitely the pattern above seems to be sth worth considering (or using the same pattern but with the pause/resume info being stored in sth like Redis or some database)