How to ensure that a flow with last task being Skipped will be considered Failed if some other important tasks fail?

Archive
, ,
1

For more information, check out the docs: Flows | Prefect Docs

View in #prefect-community on Slack

Noah_Holm @Noah_Holm: Hey all, we ran into an unexpected (to us) behaviour today when a flow was considered successful even though some of it’s tasks failed and I’m curious to hear about potential solutions.

We have this pattern of tasks in a flow with dependencies according to the arrows: task1 -> task2 -> task3
Task3 is the last task of the flow and has other upstream dependencies which might cause task3 to be skipped in a flow run. Today a flow run got the following states: Failed -> TriggerFailed -> Skipped

I assume the flow is considered successful since task3 was skipped and skipped states are considered successful (that makes sense). But we don’t want to consider a flow run successful if any tasks in the flow has a failed state. Refreshing my skills in the docs I’m thinking of handling it with state handlers, but I assume that a state handler on the flow would still see the flow state as successful due to the last skipped task. Is the only way forward here to have a state handler on each of the tasks in the flow?

@Anna_Geller: Close but no cigar! :slightly_smiling_face: instead of state handlers, you would need to use reference tasks. Here is how you can set it:

flow.set_reference_tasks(
        [
            task1,
            task2,
            task3,
        ]
    )

This way, if any of those 3 tasks fails, the flow run will be considered Failed as well.

Noah_Holm @Noah_Holm: So simple, so effective. Thanks Anna :star::star: